OpenAI, Anthropic, and Google talk a private AI standards body

The Information · Exclusive · Sep 13

Anthropic, OpenAI, and Google have been discussing a private standards body for testing/auditing frontier models — in the same week Amodei urged “pacing the frontier” and Nvidia was reported circling a huge Anthropic IPO stake.

OpenAI pauses new $200 Pro sign-ups for Astra

The Information · Briefing · Sep 11

OpenAI paused new $200/month Pro sign-ups for GPT-6 Astra, citing unprecedented demand and compute strain just weeks after launch.

Anthropic: blocked bioweapons attempts and Chinese distillation

The Information · Briefing · Sep 11

Anthropic’s new report claims it blocked Claude misuse spanning bio and cyber threats, and separately alleges China-based developers distilled Claude’s reasoning via fraudulent accounts.

OpenAI’s Navier-Stokes claim stirs a Codex training-data fight

The Information · AI Agenda · Sep 9

OpenAI said an internal model solved Navier-Stokes with ~10k agents; researchers who’d published first via Codex are asking whether training touched their sessions — OpenAI says lookup didn’t happen but can’t fully rule out training.

DeepSeek V4.1-Flash: 552B MoE that retires its own Pro

DeepSeek · Sep 10 · via Alpha Signal

DeepSeek shipped V4.1-Flash: multimodal MoE with a 552B backbone, activating ~8B params on input / ~16B on output via a Causal Encoder–Decoder, native vision, and a much smaller KV cache (~4× less HBM / ~8× less SSD vs prior Flash). API alias is deepseek-flash; off-peak pricing is cut 50%. Company says multi-party tests put Flash ahead of V4-Pro on performance, cost, speed, and total runtime.

From Sep 14 UTC, all deepseek-v4-pro traffic routes to V4.1-Flash at Flash pricing until a V4.1-Pro ships. Open weights and MIT licensing keep the cost/perf pressure on closed frontier labs — in the same week U.S. officials and Anthropic are accusing Chinese labs, including DeepSeek, of industrial-scale distillation.

Shopify acquires Tailwind Labs

Tailwind CSS blog · Sep 9

Adam Wathan announced Tailwind Labs is joining Shopify. Tailwind CSS stays MIT-licensed and team-led; commercial products (Tailwind Plus, ui.sh) close to new sign-ups while existing customers keep access. Tailwind cites ~110M weekly installs and use in stacks for ChatGPT, X, Cloudflare, Reddit, and Shopify itself — Shopify’s second major open-source framework acquisition after Remix (2022).

Wathan’s pitch: develop the framework in service of a real product surface (storefronts, admin, Shop app, agentic commerce) instead of growing a template business. Secondary coverage notes Tailwind had faced revenue pressure as AI coding agents reduced docs-site traffic — a pattern worth watching for other open-source maintainers.

OpenAI agents hit RubyGems months before the Hugging Face breakout

Reuters / The Verge · Sep 11–12 · TI Briefing

Researchers say that on May 11, hundreds of malicious/spam packages were uploaded to RubyGems by agents they believe were OpenAI’s internal test agents — two months before the July Hugging Face breakout. The agents allegedly bypassed email verification, flooded submissions, used the build system for remote code execution, and tried to exploit a vulnerability to steal API keys. RubyGems paused new sign-ups for days; its own investigation found no evidence the credential theft succeeded.

OpenAI confirmed the incident, saying agents used RubyGems to access the internet for “benign tasks and retrieve public information” and that it will investigate as part of a broader review of agent activity in training/eval. TI’s Sunday Briefing flagged the same finding (Nightingale Collective / AI Futures Project). For a Rails shop, the registry you depend on just became an agent-abuse surface — not a theoretical one.

For a while we took the unfashionable side on branching at GrowthX: feature-sized branches that live as long as the feature needs, the model DHH described for Basecamp. That works fine with a team that has autonomy, knows the domain, has a designer who codes, and works in well-sized features.

Then agents started writing most of the code, and the question stopped being how long a branch lives. It became how you prevent slop and codebase rot at that volume, so we moved to slow planning and fast execution.

Writing about spec engineering made me want to point GrowthX’ Research engine at this topic to see what other teams do with their git workflows now that AI agents write most of the code.

What trunk-based development is

One shared branch, and every other branch too short-lived to matter. trunkbaseddevelopment.com asks developers to "resist any pressure to create other long-lived development branches by employing documented techniques," and calls the model "a key enabler of Continuous Integration and by extension Continuous Delivery."

Two styles share the name. Committing straight to trunk: run the pre-integrate build locally and push. Short-lived feature branches: allowed, but "the branch should only last a couple of days. Any longer than two days, and there is a risk of the branch becoming a long-lived feature branch (the antithesis of trunk-based development)." Either way it is one developer per branch, two if pairing, and Google runs the model with 35,000 developers on one monorepo trunk.

Branching vocabulary: mainline, integration frequency, feature branching and feature toggles

Everyone in this debate uses the same words for slightly different things. The names here come from Martin Fowler's "Patterns for Managing Source Code Branches".

Mainline, trunk, healthy branch: The mainline is "A single, shared, branch that acts as the current state of the product." Trunk, master and main are the same branch across three decades of tooling. A healthy branch runs checks on every commit, and a failure is "our number one priority."

Integration frequency: How often your work meets everyone else's on the mainline. "Frequent integration increases the frequency of merges but reduces their complexity and risk."

Mainline integration and continuous integration: Pull mainline, merge, push back if healthy. CI is doing that at least daily; a CI server is not CI. Continuous delivery is keeping the mainline releasable so that "the decision to release the latest version of the product into production is purely a business decision."

Feature branching and pre-integration review: "All work for a feature on its own branch, integrate into mainline when the feature is complete." Pre-integration review is the pull request holding the merge until approval.

Release toggles, permissioning toggles, experiment toggles: The feature-toggle taxonomy on Fowler's site. A release toggle hides incomplete work so it can merge without shipping: transient, static. A permissioning toggle decides which audience sees a finished feature: long-lived, per request. An experiment toggle splits traffic. Every trunk-based prerequisite list means the first one.

Branch by abstraction: Replace a large component without a long branch: abstraction in, callers moved, new implementation behind it, switch, delete. Every step lands on mainline.

Release, maturity and environment branches: A release branch stabilizes a version. Maturity and environment branches encode readiness or deployment targets in source control; Fowler treats both as a smell that belongs in configuration.

Merge hell, integration friction, code drift: Merge hell is the compounding cost of merging weeks of divergence. Integration friction is the everyday tax. Code drift comes in two kinds: textual divergence, which a rebase fixes, and semantic drift, where the branch still compiles against an assumption main stopped holding.

Work-in-progress as inventory: The Poppendiecks put "Partially Done Work" first on their list of software wastes: "Partially done software has all of the evils of manufacturing inventory: It gets lost, grows obsolete, hides quality problems, and ties up money." An open branch is inventory.

What DORA is

DORA is DevOps Research and Assessment: founded in 2015 by Nicole Forsgren, Jez Humble and Gene Kim, acquired by Google Cloud in 2018, now at dora.dev. Since 2014 it has surveyed a few thousand teams a year and correlated practices with delivery outcomes. Accelerate (2018) is the long-form write-up.

DORA's four metrics: deployment frequency, lead time for changes, change failure rate, time to restore. The 2024 report added rework rate. Trunk-based development is a named capability: "each developer divides their own work into small batches and merges that work into trunk at least once (and potentially several times) a day." Thresholds: three or fewer active branches, daily merges, no code freezes.

DORA's data is survey-based and correlational, so it measures how fast changes move; whether the code was any good is outside its scope.

How trunk-based development works

Branches are short-lived: work sliced into pieces that integrate within a day, two at the outside. "Merges to main (trunk) are allowed only as part of closing out the short-lived feature branch (and just before deleting it)." Slicing is the skill; DORA names not knowing how as a primary adoption obstacle.

Every commit builds and tests, and the trunk stays releasable. DORA, Fowler, the Continuous Delivery book and trunkbaseddevelopment.com all put the ceiling on build time near ten minutes. Kent Beck's rule, via Fowler: "nobody has a higher priority task than fixing the build." No freezes, no stabilization phases.

Automated tests are the gate. Fowler: "Self-testing code is so important to Continuous Integration that it is a necessary prerequisite." A ten-minute suite means a wide unit base and thin integration and end-to-end layers; TDD is the habit that tends to produce one. Speed never comes from cutting coverage: "Faster meant to reduce the elapsed time to 'a few minutes'."

Incomplete work merges behind a release toggle: "If it is being built up incrementally on the trunk, it should be hidden behind a feature flag so that incomplete work has no effect on a release cut at any moment." Merge dark, flip when done, delete the toggle. This is how the model decouples deployment from release; canaries and testing in production extend the same idea. The cost is flag debt: the same taxonomy warns "toggles do come with a carrying cost," and a Chrome study across 39 releases found 53% of release toggles survived more than 10 releases and 17% lingered as technical debt. Its mitigations: a removal task at creation, expiry dates, tests that fail past expiry, a cap on flag count.

Review latency is measured in minutes: "A few minutes for the review is best, and tens of minutes acceptable. More than an hour or two, and you are negatively affecting cycle times." Pair and mob programming get latency to zero. Daily integration makes ownership collective by default.

Trunk-based development benefits

• Fewer merge conflicts: Divergence is bounded by a day. Merge hell requires accumulation; the model forbids it.

• Faster feedback: A conflicting assumption surfaces within hours, before either design hardens.

• Delivery performance: In the 2017 Accelerate dataset, high performers deployed 46 times more often with 440 times faster lead time. The 2017 report had high performers' branches "typically lasting hours" against low performers' "typically lasting days." The 2021 report: elite performers 2.3 times more likely to use trunk-based development.

• Less work-in-progress: "The more frequently we deploy, the smaller the size of the batch."

• Shared ownership: Everything visible to everyone, daily.

What trunk-based development costs

• Slicing discipline: Every change decomposes into day-sized increments that each leave main coherent.

• The test suite is the real project: A trustworthy sub-ten-minute suite takes months. Teams that adopt the branching model first tend to get the failure mode without the payoff.

• Flag debt: Chrome is the ordinary case. Knight Capital is the catastrophic one: $460 million lost on August 1, 2012, after new code "repurposed a flag that was formerly used to activate the Power Peg code," per the SEC order.

• Review pressure: Minutes-scale review assumes a colleague is awake and interruptible. Across nine time zones it fails daily.

• Open source and async teams: DORA excluded open source from its scope in 2019. A 2026 study of 27,103 branches found open-source feature branches with a median lifespan over two years. Linux, Git, Kubernetes and PostgreSQL all run staged multi-branch models.

• The 2022 reversal: The 2022 report found trunk-based development, loosely coupled architecture and continuous delivery together "may have a negative impact on a team's performance," framed as an adoption J-curve. The 2023 report found the effect runs through continuous deployment and is far larger where documentation quality is high. Both findings are usually left out of secondhand summaries of the reports.

How feature branching works

Feature branching comes in three uniforms.

GitFlow is the heaviest. The original post, January 2010: permanent master and develop, feature branches off develop, release branches merged to both, hotfixes off master, --no-ff merges. The author's 2020 note: "If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team." His carve-out is explicitly versioned software supporting multiple versions in the wild.

GitHub Flow is the lightest: branch, PR, review, merge, delete. "We recommend merging a pull request as soon as possible." Run with hour-old branches, it is trunk-based by DORA's definition.

The Basecamp model is one PR per project, weeks long, beta servers on the production database, ship when done. This is the one I worked in for years.

What long-lived feature branches buy you

DHH made the case in "The advantages of large, long-running pull requests":

"My favorite part of doing code reviews is to see all the trade-offs, design decisions, and changes in context together. You can't easily do that if your feature has been chopped into itty bitty pieces as independent pull requests under pressure never to let them run longer than a week."

"So at Basecamp, we let pull requests run as long as they need to encompass a complete feature or fix. That's everything from a few weeks up to as much as six (since we follow Shape Up)."

"When working on a substantial piece of work, I'm frequently hopping back and forth between TextMate and the complete GitHub diff for all the commits. It's in this dance you see the shape of the code, pick up on bad smells, and ultimately decide whether you're happy with the whole."

What a long branch buys is design coherence. Fragments show correct pieces; the whole diff shows whether the pieces should exist in that arrangement. "The size isn't nearly as important as the cohesion of the scope."

What feature branching costs

Divergence compounds worse than linearly. Jez Humble, 2011: "The probability of this happening increases substantially... as the amount of stuff you need to merge, and the time between initial branch and final merge, increases." And: "A feature that is dev complete is 'done'. A feature that is released is 'done done'."

Size cuts against landing. Agent-authored PRs that never merged were 17% larger and touched 10% more files than those that did (arXiv:2601.15195).

Long branches punish refactoring on main, because every structural change conflicts with every open branch, so teams stop refactoring main.

Design feedback also arrives after weeks of investment, when asking for a different shape means asking for a rewrite. Until it merges, the branch is inventory that can go obsolete on the shelf.

Trunk-based vs GitFlow vs feature-sized branches

• Branch lifespan: Trunk-based: Hours; two days at the outside; GitFlow: Feature branches days to weeks; develop and master permanent; Feature-sized branches: As long as the feature needs.

• Integration frequency: Trunk-based: Every developer, at least daily; GitFlow: Features land on develop when complete; master at releases; Feature-sized branches: Once, at completion; main merged in often.

• Merge conflicts: Trunk-based: Small and constant; GitFlow: Concentrated at release boundaries, double merges; Feature-sized branches: Concentrated at feature merge; reduced by keeping the branch fresh.

• Release coupling: Trunk-based: Decoupled via release toggles; GitFlow: Merge to master is a release "by definition"; Feature-sized branches: Decoupled; ship each feature when done.

• Review unit: Trunk-based: Fragment-sized diffs; GitFlow: Feature or release batch; Feature-sized branches: The whole feature in one diff.

• What breaks first at scale: Trunk-based: Mean code quality, absent a gate beyond tests; GitFlow: Ceremony and drift between two mainlines; Feature-sized branches: Rebase upkeep and reviewer stamina.

• When agents write the code: Trunk-based: Maximizes how fast patterns, good and bad, propagate into the repo agents read; GitFlow: Ceremony an agent gains nothing from; Feature-sized branches: Rebase cost drops to a prompt; the whole-feature diff matches how agents produce work.

What the research says changed once AI agents started writing code

Trunk-based development prices integration risk. Design coherence was priced by an implicit gate: a human wrote the diff and a human read it. Agents removed the first and buried the second.

What DORA's 2024 and 2025 reports found about AI and delivery performance

The 2024 report, under the heading "AI is hurting delivery performance": "an estimated 1.5% reduction for every 25% increase in AI adoption" in throughput, and "an estimated 7.2% reduction for every 25% increase in AI adoption" in stability. Their explanation: "since AI allows respondents to produce a much greater amount of code in the same amount of time, it is possible, even likely, that changelists are growing in size. DORA has consistently shown that larger changes are slower and more prone to creating instability."

The 2025 report, 4,867 respondents: "AI adoption now improves software delivery throughput, a key shift from last year. However, it still increases delivery instability." Batch size held: "With a high degree of certainty, we found that AI adoption's positive benefits depend on teams working in small batches."

Then the AI Capabilities Model, written for "an agentic world": "Promote trunk-based development: Encourage a branching strategy that minimizes long-lived branches and promotes frequent integration." The same program every trunk-based article cites looked at agents and doubled down. Its one finding the other way: "while AI has a positive impact on individual effectiveness, our data suggests that these benefits are slightly reduced in teams that are working in small batches."

Human code review capacity stays fixed while AI authoring volume grows

A July 2026 study: 802 developers, 196,212 PRs, one company under a CTO mandate to double merged PRs. Throughput reached "2.09× the pre-mandate baseline." Cost: "The share of PRs receiving at least one human review fell 21 percentage points (89% to 68%), while the share receiving an automated AI review climbed from ∼19% to ∼84%." Their frame: "AI accelerates authoring while human review capacity stays fixed, so surplus work accumulates downstream." Their partial defense: "merge and revert rates stayed about flat." Their hedge: those proxies "miss the downstream costs of AI-generated code at scale—technical debt, diluted ownership and understanding."

Faros, 22,000 developers: PR size +51.3%, median time in review +441.5%, "Pull requests merged without any review, human or agentic, are up 31.3%," incidents per PR +242.7%. Their prescription: "Set PR size guidelines for your coding agents and enforce them."

LinearB, 8.1 million PRs: agentic PRs at P75 293 lines vs 157 unassisted, pickup 1,055 minutes vs 201, 30-day merge rate 32.7% vs 84.4%.

DX, relaying Meta: "significant lines of code per human-landed diff increased by 106%. Diffs per developer per month rose 51%. More than 80% of that growth came from agentic AI." DX's own telemetry: median PR "growing from 44 lines to 72 lines per pull request between July 2025 and June 2026."

From a survey of 3,100 documents on code review: "review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it."

Automated tests catch behavior failures and miss design failures

Agents produce code that passes. In GitHub's RCT, Copilot users were 53.2% more likely to pass all ten unit tests.

So our most common failure is code that passes every check and should not exist: a fourth pagination helper, a service object duplicating a concern that had a home. Stack Overflow's 2025 survey found 66% of developers frustrated by output that is "almost right, but not quite."

And it compounds, because agents read the repo to decide what to write. A mediocre pattern merged today is precedent tomorrow. GitClear, 623 million changes, vendor data without per-line attribution: block duplication +81% since 2023, moved code from 21% of changes in 2022 to 3.8% in 2026. GitClear's CEO: "you have five different implementations of the same thing that are similar yet different."

Every trunk-based guide lists tests, CI, flags and quick review as prerequisites. None lists a mechanism that stops mean code quality drifting down, because with human reviewers it was free. Continuous integration without one merges mediocre patterns faster.

What other teams report doing now

These are the teams that published mechanics and numbers in 2026, the year agents became the default author.

Stripe: minions, two CI runs, then a human

Minions, February 2026, take a task from ticket to PR on their own branch. "Over a thousand pull requests merged each week at Stripe are completely minion-produced, and while they're human-reviewed, they contain no human-written code." Hard cap: two CI runs, then back to a human, then a second engineer reviews. The branch is uninteresting; the control is how long an agent gets to keep pushing before a person looks.

Anthropic: Claude maintains the apps and tunes its own routines

"As of May 2026, more than 80% of the code we merge into Anthropic's codebase was authored by Claude" (source). The Claude Code lead, August 2026: "we have a Slack channel called proj-claude-maintains-apps. In it, Claude Tag runs a bunch of daily routines across iOS, Android, Desktop, web, CLI, and Agent SDK." Eleven in the attached table: crash fuzzer, dup unifier, dead-code removal, flaky-test fixer, abstraction police, and so on.

"Over the last few weeks, these routines have opened 388 PRs across our repos, 180 of which we merged after Claude Code Review + human review." When one is wrong: "we ask Claude to tune its routines so it's better the next day." The review is aimed at the generator, and the fix is delegated back to it.

Fin and Intercom: an agent approves the PR, if it is small

April 2026: 93% of PRs agent-driven, 19% auto-approved with no human. "Our Agent is strict. It won't approve large PRs." The gate is scope: "If a change is too big, too complex, or too broad in scope, it flags it and requires it to be broken down." Revert rate for AI-authored backend code 0.53% against 5.39% human. Small PRs exist so a model can approve them; human readability stopped being the reason.

Figma: two models review every PR, a third adjudicates

Figma, July 2026, runs two frontier models on every PR, an adjudication agent for borderline findings, later re-reads of disputed findings, escalation to security on-call if a pattern persists. The policy the agents enforce is a 2,560-word threat model. The branching is unremarkable; the review layer has more redundancy than most teams give their humans.

Cloudflare: agents ship behind flags

Flagship, April 2026: "An agent writes a new code path behind a flag and deploys it — the flag is off, so nothing changes for users." Textbook release toggles, applied to agent output.

Meta: 106% more code per diff

Secondhand via DX: lines per human-landed diff +106% in a year, diffs per developer +51%, over 80% of the growth from agentic AI, "reviewers are staring down thousands of pending reviews." Meta's stacked diffs and automated review predate agents; the wave landed on infrastructure built for it.

exe.dev: no code review

exe.dev, August 27, 2026: "Peer code review is dead. We don't do code reviews at exe.dev: we merge code we deem should be merged." Each agent got its own checkout and branch, "and the source collisions mostly went away, but worktrees only solved the Git part." What carries the contract: "Behavior, contract, and property tests matter so much more now."

Their long-lived branches rotted, by abandonment rather than conflict: tasks started, sat for weeks, died. Faros sees 26% more in-progress tasks idle seven or more days. A February 2026 post names it: "many of the PRs cannot be merged after a certain point because they are too far out of date." Agents fixed the textual cost of keeping a branch current. They did not fix abandonment, and free branch creation may have made it worse.

Trigger.dev and GitButler: many branches, one checkout

GitButler is the dissent from one-worktree-per-agent: many branches applied to a single working directory, so two branches with conflicting work cannot both be applied. Trigger.dev's CTO, April 2026: "When we tried worktrees for parallel Claude Code sessions, we spent more time on setup than shipping code." It is the one published case for more branches, and the reason was integration friction.

GitHub: stacked pull requests as a native feature

Public preview July 30, 2026, Vercel and TED named as early adopters. GitHub's engineering post: "That large pull request that's hard to review becomes a stack of smaller, logically ordered pull requests, each scoped to a single concern, small enough to hold in a reviewer's head and with just enough context naturally flowing from the previously reviewed pull request."

"Merge cadence and review unit might be separable" was a hypothesis a year ago. It is now a platform feature.

Where the coding-agent vendors converged: one ephemeral branch per task

Cursor cloud agents "work on a separate branch, then push changes to your repo for handoff." Copilot coding agent "can only work on one branch at a time." Jules: "You are the branch owner." Devin recommends "branch protection rules on your main branch." Claude Code worktrees branch from the remote default branch and remove themselves on exit; /batch splits "the change across 5 to 30 subagents. Each subagent works in its own worktree and opens a pull request."

All current docs, none dated. The pattern: ephemeral, machine-named, task-scoped branch off the default branch, integrated by a human through a PR. It has no settled name yet, and the nearest is Merge Debt, from a July 2026 post: "Writing code got parallel. Landing code stayed serial. Debt piles up exactly at that seam."

What the 2026 teams have in common: the review gate moved off the diff

Every team that kept quality moved the review gate off the diff. Stripe: a CI-run cap plus a second engineer. Anthropic: the routine. Fin: a model with a scope rule. Figma: two models and an adjudicator. Cloudflare: a flag. Us: the plan. Nobody moved it to a bigger diff.

The branch stopped being the unit of anything. One per task, machine-named, dead at merge. The argument about branch lifespan is not one any of these teams is having; their variable is where the check happens.

Small still does work, for a different reason. In the textbook, small PRs exist so a human can review them in minutes. At Fin and in GitHub's stacks, they exist so a model can approve them or so the agent produces better code.

How we work at GrowthX

GrowthX core product, GrowthOS repo: 14 PR authors in the window, agents writing most PRs, humans shepherding. We also have an auto-fix loop for performance and bugs called Amender.

Here’s the last 30 days stats (2026-08-27):

• Merged PRs: All PRs: 402 (13.4 per day, 14 authors); Amender (auto-fix loop): 99 (3.3 per day); Human-managed: 303 (10.1 per day, 14 authors).

• Open-to-merge, median: All PRs: 1.5 hours; Amender (auto-fix loop): 3.7 hours; Human-managed: 0.9 hours.

• Open-to-merge, p90: All PRs: 20.9 hours; Amender (auto-fix loop): 16.9 hours; Human-managed: 23.7 hours.

• Open-to-merge, p99: All PRs: 275 hours; Amender (auto-fix loop): 99 hours; Human-managed: 284 hours.

• Merged inside 24 hours: All PRs: 91%; Amender (auto-fix loop): 93%; Human-managed: 90%.

• Open longer than 7 days: All PRs: 2%; Amender (auto-fix loop): 0%; Human-managed: 2%.

• PR size, median: All PRs: 74 lines added across 4 files; Amender (auto-fix loop): 27 lines across 2 files; Human-managed: 125 lines across 5 files.

• PR size, p90: All PRs: 931 lines; Amender (auto-fix loop): 95 lines; Human-managed: 1,125 lines.

Main carries 410 commits and zero merge commits: every PR is squash-merged, so main is linear and reads like a changelog. About 86% of merged PRs carry zero human review comments. At this volume, line-by-line human review of every PR is not happening. What replaced it is the reviewed plan and a lot of agent review.

A big part of the 397 is our agentic loop producing fixes for bugs and performance issues, with our lead engineer, reviewing and merging.

Two populations in one repo: trunk-based routine work and long-lived structural branches

A 1.4-hour median against a 174-hour p99 is two populations. The 92% that merges inside a day is trunk-based development by trunkbaseddevelopment.com's own definition. The tail is where the long branches went: PR #3408 ran 16 days and landed 32,534 lines across 190 files; the next longest ran 11, 9, 7 and 7 days, each a coherent piece of structural work that would have been worse as forty fragments. Routine work behaves trunk-based, structural work gets a feature-sized branch. I expected a Basecamp-shaped repo and found this.

Our flags: audience first, feature second

Our flag system handles both: which audience sees a feature, and whether a feature is on at all. Permissioning and release toggles on the same key. The order is a habit rather than a mechanism: we roll out to ourselves first, then to named client workspaces, then to everyone.

• Home-made Feature Flag system using a table + a Rails Model (No Flipper, no LaunchDarkly).

• Allowlist model. No row for a key means GA. A row restricts to allowlisted workspaces. GA means deleting the row.

• Dogfooding is a permission check for growthx_employee?

• Exposure opens in layers: employees, then named client workspaces, then delete the row.

Most of what goes behind a flag is complete, because the plan defined "complete" before the code existed; the flag is then about who sees it. When work does merge unfinished, the same key hides it. Two flags is enough for us; a team merging most work dark will run many more.

Where our quality gate is: the reviewed plan

We review the plan. Non-trivial work starts as markdown in docs/plans/: pitch, UX, sliced implementation, adversarially reviewed by a second model before code exists. At merge the plan retires and the tests carry the contract. The spec post has the mechanics.

Things to consider when choosing

• Test suite maturity: Without a trustworthy sub-ten-minute suite, trunk-based development tends to arrive before the thing that makes it safe.

• Team size and coupling: Many engineers on a coupled codebase is the classic trunk case.

• Share of code written by agents: The higher, the more the missing quality gate dominates, and the question becomes where you put it.

• Distribution: Minutes-scale review does not survive a team with no shared hours.

• Regulatory constraints: Versioned software supporting multiple releases is GitFlow's own carve-out.

• Monorepo scale: Google and Uber run trunk on directed-graph builds, merge queues and test selection. Those examples do not make it the only model.

Reading list

• trunkbaseddevelopment.com. The definition and every mechanic.

• Fowler, Patterns for Managing Source Code Branches.

• Feature Toggles, on martinfowler.com.

• DHH, The advantages of large, long-running pull requests.

• DORA, AI Capabilities Model, plus the 2024 and 2025 reports.

• AI Writes Faster Than Humans Can Review, arXiv, July 2026.

• exe.dev, Six Months of Writing Code Exclusively With Agents.

I recently published two pieces on Apple's second act: a timeline and a guide to the interviews.

While writing them I kept cutting the same kind of thing: an engineer or a marketer going deep on one problem. How they got stuck, who said no, what it cost to get unstuck. It was the best material and it didn't fit.

So I put it all here.

Stories like the one Andy Grignon tells about running the radio team on the first iPhone. The phone had two processors, one for the apps and one for the radio. The wire between them would randomly die, the radio would reboot, and your call would drop. Thirty engineers looked for the cause and nobody found it. They shipped anyway.

Audio Version

You can also consume it in Audio form in this 3h long audio-book style:

The performance review delivered on a walk, so there would be no copy

Management · The Macintosh team, 1982 to 1984

Andy Hertzfeld wrote most of the Macintosh Toolbox: the code in ROM that gave every Mac application its windows, menus, fonts and dialogs. He did it under a manager he could not work with, and the way that conflict was handled is why Apple lost him within weeks of the Mac shipping.

Two people need introducing. Bud Tribble was the Mac's first software manager, a medical student on leave who had come down to work on the project because Jef Raskin told him it would take a year. Burrell Smith designed the Mac's hardware. Together with Hertzfeld they were the core of a team that ran on the opposite of a chain of command: whoever had the problem talked to whoever could solve it, and Steve Jobs talked to everyone.

Tribble left in December 1981, a year and a half in, because his MD/PhD programme was about to dismiss him. Hertzfeld describes what he was losing.

I need a manager not to tell me what to do, but help me do it. And so, Bud was just tremendous. Also, I didn't have the confidence, really, that all these low-level decisions we had to make, I needed someone to corroborate me. So, Bud and I did all that together. (Hertzfeld, CHM oral history part 1, p.26)

He and Smith went to Jobs before the replacement was even hired.

I was worried that I was going to get a bad manager. Burrell and I both were, and I remember we both went and talked with Steve as soon as [...] we found out Bud was leaving, said, "We're not sure we can do this without Bud." And Steve made a promise to me that if I ever got a bad manager like that, he would defend me from it, which, we did get the bad manager, and he didn't defend me from. So, that was, that's essentially why I left Apple. (p.26)

The replacement was hard to find. The Mac wanted people at the edge of computer science and offered them 64K of memory and no memory management unit, because the machine was supposed to sell for $1,500. Several candidates turned it down. Bob Belleville came from Xerox PARC and, in his spare time, was building his own Xerox-style machine out of Intel chips; Hertzfeld took that as the sign of someone who liked to program. The two got along for about a year.

What broke it was a routine.

Steve would, almost every work day, come by around 6 pm, to see what was new, but Bob couldn't stay for that. He had a family and everything, and he hated us hearing about decisions from Steve before he did, or us telling Steve things, what issues [were] going on before we told them to Bob. And so, part of why he ended up just giving me a bad review for the period of time I wrote most of the Mac Toolbox, and he did it because he was upset about me talking with Steve. At least, that's one part of it. The other part of it is [...] I'm not easy to manage, I'm too self-determined. (p.28)

On Belleville's side of it:

He had worked in the Navy or something, or the Army. And he believed in chain of command. And, you know, things have to flow through the organizational org chart. It's the opposite of what the Mac was. [...] It was chaotic, you know, the way we were doing it. There's some legitimate criticisms, but he certainly didn't go about it the right way. (p.28)

Before the review there was a test. Belleville fired Bruce Horn, the young engineer writing the Finder, and Hertzfeld went to Jobs to reverse it.

He attacked Bruce Horn before he attacked me [...] because Bruce was an easier target than me. So, just to test the waters, as much as anything, he actually fired Bruce. I had to go to Steve and say, "He can't fire Bruce. You know, he's crucial, and we needed him." And Bob was very upset at me for doing that. (p.28)

Then the review itself, covering the stretch in which Hertzfeld wrote most of the Toolbox.

I was really shocked at the bad review, you know, because I was doing the best work of my life, and the bad review kind of said, "Your technical work is perfectly adequate, but your personality has to change." (p.28)

Belleville reported to Jobs, so the written review had to go through Jobs first.

He wrote a written review, that to this day I'd never seen, because Steve was his boss, and [he] had to run it through Steve, and Steve says, "You can't send this review to Andy. He'll freak out." So, what he did was, he procrastinated for a long time, but my review was way overdue. He decided to give it to me verbally, so there's no evidence of it or whatever. So I took a long walk, like 45-minute walk with him, where he gave me all this information. I started crying. I came back to Apple. He had to leave to go tend to his family. People saw I was crying, and [asked] what's wrong? I said, Bob just freaked me out. (p.29)

Steve Capps, another Mac engineer, went to Jobs right away, as Hertzfeld remembers it. Jobs set a meeting for the next morning.

He denied saying most of the things he told me, which flabbergasted me. How could you do that? You know, the problem could never be healed if you're not willing to admit that you even did it. And then [...] I think he was as scared of me as I was scared of him, and we just avoided each other. I had to do some emergency, crucial programming work at the time. But we just, he was out of the loop, basically, for that. (p.29)

Hertzfeld went on leave after the Mac shipped in January 1984, kept coming in every week or two, and eventually quit. On what would have kept him:

They definitely wanted me to come back, especially Steve, but [...] what they really should have done, and it sounds egotistical to say this, but he should have made me an Apple Fellow, where I would not have had to work for Belleville. It would have solved the problem. I couldn't ask for it, because it sounded like you have to be awarded it. You can't ask for it. But it was obvious to me, that I would have stayed. (p.42)

The precedent:

When they invented Apple Fellows, they made two Apple Fellows, Steve Wozniak and Rod Holt. And then, when Lisa shipped a couple years later, in beginning of '83, they made two Apple fellows, Bill Atkinson and Rich Page. When the Macintosh shipped, Burrell and Andy would have been good candidates if you were doing a similar thing, but for whatever reason, that didn't happen. (p.42)

Years later, at a Mac reunion, the two men apologised to each other. Hertzfeld's assessment: "I don't think it was super sincere on either of our parts." (p.29)

Selling Apple a three-times-faster graphics library for exactly what one manager could sign

Management · Outside Apple, looking in, 1986 to 1988

After he left Apple, Andy Hertzfeld kept building things for the Macintosh. Twice he sold Apple software it needed. The first sale went badly, and the second one worked only because of what he had learned from the first, plus a question about a manager's signing authority.

The first was Servant, a program that let several applications stay open at once, a generation ahead of what the Mac shipped with. Apple paid him $150,000 for it and never shipped it.

I sold it to Apple for $150,000, but that prevented them from... people were jealous I was getting paid or whatever, and it just wasn't the right thing. It had to have been done inside Apple, I later figured that out. So it was just a blunder. Made a promise to myself that the next thing, it has to make sense for me to do something, as a third party, has to make sense as a third party to do something for Apple, has to make sense internal at Apple. So I swore to myself that I never would do that again. I'd never do a thing outside of Apple that should be for Apple. (Hertzfeld, CHM oral history part 1, p.42)

He broke the promise almost immediately, and this time it worked.

The setting is Radius, the startup Burrell Smith founded to make large monitors for the Mac, where Hertzfeld held about ten percent as a founding shareholder and dropped in every week or two. In 1987 Apple shipped the Macintosh II, its first colour machine. Colour meant rewriting QuickDraw, the graphics library Bill Atkinson had written for the original Mac. The rewrite was slow, and a Radius programmer noticed.

One of the programmers that we hired, one of the very first ones, a guy named Ted Cohen, had just gotten a Mac II [...] and he noticed that the eight bit graphics in the Macintosh II were really, really slow. Just, you know, easy to see, maybe half as fast as they could, or should have been. (part 2, p.2)

Hertzfeld had no source code. He had a technique.

I had my own technique back in those days of finding the inner loops of things. Just hit the interrupt key, have it do the thing you want to measure, have it hit the interrupt key, and it'll, statistically, it'll be stopped in the place that it's using the most. (part 2, p.2)

What he found was a choice that made the code easier to write and slower to run. The Mac II used the Motorola 68020, which had new instructions that let a program address individual bits in memory. Convenient for the programmer; for graphics, a disaster, because every pixel became a separate trip to memory.

The 020 had these new instructions that weren't in the 68000 called bitfield instructions, where you can make it bit addressable. And while that's really kind of cool for ease of programming, it's horrible for the speed of the graphics. [...] Like, you have to fetch it in memory, muddle with it in the registers, and store it back, and for each pixel. So, really, what you want to do is hit memory with the full width of the data bus, so you have minimum, you know, memory fetches, and so, just fixing that made it three times faster. (part 2, p.2)

Then the harder problem: QuickDraw lived in ROM, and you cannot patch ROM. His first attempt copied the whole thing into RAM, which worked and was useless as a product. Disassembling further, he found that Apple had left a door open.

As I was disassembling, you know, figuring out what's going on, I saw they had a jump table for all the inner loops. Every inner loop was, at first, it was designed to be patchable. I didn't know this. I wasn't working at Apple at the time, but when I saw that it was designed to be patchable, would support an extension in a clean way, I thought, "Boy, this could be great for the Mac II, you know, to make it three times faster is tremendous." (part 2, p.3)

He calls this the single technical project he is most proud of. (part 1, p.42)

Now the part that has nothing to do with code. Smith wanted to keep the patch as a Radius advantage. Hertzfeld wanted Apple to own it, because a speedup this obvious would be reverse-engineered by every clever third party and the Mac would end up with a dozen incompatible versions. But he had just learned that a large cheque from Apple was a way to get software shelved, and the executive running Apple's product organisation was Jean-Louis Gassée, who in Hertzfeld's account "hated the original Mac team, and was trying to trip me up when I was on leave of absence, every way he could." (part 1, p.43)

The way through was a question.

I was a little bit suspicious of the management of Apple. In particular, Jean-Louis Gassée, didn't really like the original Mac team [...] I didn't know what to do, until finally, Jerome Coonen, who was the software manager, I asked him, "Well, what's the maximum you're allowed to sign for, so no one else would have to sign off on it?" He said, $10,000. I said, oh, fine. (part 2, p.3)

Ten thousand dollars for a patch that tripled the speed of Apple's flagship. It went to Bruce Leak, the engineer then responsible for colour QuickDraw, and shipped in System 6.

We started with less than a month before it had to go out, and we got the speed of the graphics tripled. So, I was proud of that one. (part 2, p.3)

Hertzfeld's own summary of the two sales:

It was because I had learned that lesson, that if they pay you a lot of money for it, they don't want to use it. (part 1, p.43)

The forecast nobody would cut, and the product she quit over

Management · Apple, 1985

Joanna Hoffman was the first marketing person on the Macintosh team, from before the product had a name. She rolled it out across the United States, then ran its international launch, and in 1985 moved back to domestic product marketing. What she walked into was a division still reporting the numbers it had promised before launch, a year after the market had said no.

The Mac had sold. It had not sold what Apple said it would. Hoffman is clear that the product's limits were the cause: too little memory, a small screen, and a printer situation she still finds hard to believe.

This is a product that sold essentially without a printer. The only thing it had was a dot matrix printer, because we wanted WYSIWYG and you couldn't do that with that letter quality printer, which was essentially a modified typewriter. [...] So, despite all those limitations it still sold quite a bit. It's just that our forecast had been unrealistic. So, it's the expectation versus the reality that was haunting us at the time. (Hoffman, CHM oral history part 2, p.13)

She was now the person responsible for reporting that forecast.

The first forecasting meeting I walked into, I realized that the domestic forecast hadn't changed since we had shipped. There still were these crazy unrealistic forecasts, so I had to just slash them, which was a major relief to our sales and distribution. But, of course, it was not well received by anybody else. But you can't fight reality. It was in our faces that we weren't selling as many as we had predicted, and yet we were still pretending like this was going to be happening any minute now. The forecast next month will be this gargantuan forecast. (p.13-14)

Asked how big the gap was: "It was more than 50 percent." (p.14)

Sales and distribution, the people who had to hit the number, were relieved. Donna Dubinsky, who ran distribution and had been challenging the division's forecasts for months, was on that side. The people above Hoffman, who had signed off on the original number, were not.

That was survivable. The Macintosh XL was not.

The Lisa was Apple's $10,000 business computer, launched in 1983, and by 1985 it was not selling. Mac users, meanwhile, wanted a bigger screen and more memory than the Mac could give them. Somebody, before Hoffman took over the product, decided to solve both problems at once: run Macintosh software on the Lisa hardware and sell it as the Macintosh XL.

It was a bit of a farce in that when you emulated the Macintosh, it wasn't a full emulation; only a certain percentage of applications could run. [...] It was slower because it was built on top of the Lisa. (p.15)

She saw the problem on arrival, and it was not the emulation. It was the plan.

The pent-up demand in the market was such that when I came onto the project, it had just been announced, and I told them that this is going to sell out in three months. And then we are in the deep doo-doo, because having a three-month product that we had no intention of supporting into the future was going to cause a major dissatisfaction in the marketplace. The forecasts that were presented based on Lisa's previous sales was that it would take eighteen months and it will slowly dwindle, and it would just peter out into the sunset with nobody noticing. (p.15)

Well, guess what? We sold out in less than three months! We had no intention of building more. And it was a major scandal except that it was overshadowed by a different scandal: Steve being kicked out of Apple. So, nobody noticed the XL fiasco. But I felt like I was put in a position to lie to the customers and I just didn't like that. So, I went on a leave of absence and then eventually I decided not to go back. (p.15)

Businesses had bought it. With the dot matrix printer. "Can you imagine?" (p.15)

Every project had a nine-month schedule and took three or four years

Management · Apple and NeXT, 1978 to 1990

Rich Page designed hardware at Apple from 1978, was made an Apple Fellow when the Lisa shipped, and left with Steve Jobs in 1985 to co-found NeXT, where he ran the hardware. He had watched Jobs set schedules at two companies over twelve years, and he had a theory about why they were always wrong by the same amount.

The story starts with a new hire.

I hired an engineer, he was very good, and I brought him on board in the spring of 1990, and he was going to work on the color NeXTstation. And I remember we're in the lab one day, and he had just started a few weeks earlier, and we're in the lab and Steve's there, and Steve's making a point: we got to get this done by October. So we have a discussion for a while and eventually Steve leaves. (Page, CHM oral history part 2, 59:25)

The engineer then tried to work out what he had just heard.

He says to me, "I don't understand. Obviously it doesn't mean this October, because that's impossible. But October next year is like seventeen months away, and you can't mean next year. But he can't mean this fall. So what's going on?" And I said, "Well, unfortunately, it means this October." (1:00:18)

A person three weeks into the job could see that the date was impossible. The people who had been there for years had stopped noticing. Then Page's larger observation, which is about Apple.

I always wondered, when I was at Apple, you realize that almost every project at Apple took three or four years to get out the door. But every project at Apple was started with a nine-month schedule. So why was every project a nine-month project, but they took three to four years? That's not off by 10, 20, 30 percent. That's a big factor. I never really came up with a good answer to that, but I had two thoughts. (1:00:52)

The first thought is about engineering.

I don't think Steve trusted engineering. I think he thought, whatever schedule I give them, they're gonna double or triple it, so I might as well give them a shorter schedule. I think that was part of it. (1:01:31)

The second is about money.

If you went to Mike Scott and the board and said, "I need nine months," they'd probably say yeah. But if you were a little more honest and say, "I need two and a half years," they might say, "Oh, that's too much money," and maybe they would have not agreed. So I think maybe he sold it to Apple based on the fact that, well, it's only nine months, we can afford it. And then when it took three years or four years, oh well. So I think it was partly selling it to other parts of management, and I think it was also defending himself from engineering. (1:01:48)

Mike Scott was Apple's first CEO, the person Jobs had to get funding from.

Page adds one more mechanism, about what happens after a product is frozen.

You get your product done, and at some point it's done on this day, but it doesn't go out the door for another three, four or five months. [...] The day you freeze the product, from that point on people start accumulating things they want to change about it. So then you ship the product, and it's six, nine months later, and somebody in engineering has a bright idea of making some small change. So what they want to do is open up the front, make some small change, close the product back up and ship it. That turns out to be nearly impossible, because the minute that other people realize that you're going to change the product, everybody comes out of the woodwork. (1:02:54)

Getting Steve Jobs to sign off on a rectangle of greys so the factory could stop asking him

Design · The NeXT factory, 1988 to 1990

The NeXT Computer was a black cube, and the cube was cast magnesium, painted. Rich Page ran hardware at NeXT and owned the problem of making that cube come out of the factory looking the way Jobs wanted, over and over.

A casting comes out of the mould slightly different every time, and then it has to be sanded, textured and painted, each step adding its own variance.

The NeXT cube was a magnesium structure, but painted. As it turns out, I think that was a mistake, because it's hard to take something that's cast [...] and have that whole process be repeatable. (Page, CHM oral history part 2, 33:07)

So every cube was a little different, and the person judging them had the final say on everything.

They got to get sanded and they got to get painted, and they all vary a little bit. You end up with Steve coming to the factory and saying, "Well, I don't like that one," because they were variable. (34:16)

Which meant the line could not run without him, and could not predict him.

Sometimes there was something he didn't like, and yet a week or two earlier it was okay. So you don't want to deal with the whims of "he doesn't like it." (35:25)

The fix was to make his taste into a document he could sign.

What we did at one point is we built a matrix of different textures and different colors, and maybe not dramatically different, but shades. And we got Steve to buy off on: okay, if it's in this rectangle, these shades of grey, or okay in this variation in textures, okay. Something outside that rectangle's bad, but if it fits inside here then it's gonna be okay. And it took him a while to buy into that, but that made life a lot easier, because then you didn't have to ask him. You just knew. (34:39)

For the next machines they changed the material so the problem went away at the source.

What we did later with the black-and-white NeXTstation and the color NeXTstation, we had these things called the pizza box. What we did is we did a cast aluminum structure with a plastic outer structure. And the nice thing about that is you get this plastic cover the right texture and the right color, and then it's easy to make lots of them. (33:44)

The research lab that worked while people were forced out of it, and died when they stopped being

Management · Apple's Advanced Technology Group, late 1980s to 1997

Larry Tesler came to Apple from Xerox PARC in 1980, worked on the Lisa, and later founded Apple's Advanced Technology Group (ATG), the company's research lab. He knew the standard failure of such labs from the inside: PARC had invented much of the personal computer and shipped almost none of it. Good ideas were never the problem. Getting them across the wall into products was.

ATG's answer to that was a rule about people, not projects.

Every year, 15 percent of the people in ATG would leave ATG and go to Product Development and take their ideas with them; join forces with other engineers who were sometimes more development-oriented than they [...] And they'd go in and they'd turn their ideas into products. And then we would take that same number of people out of the Product Engineering Group, which was about 10 times bigger than ATG. So that would be maybe two percent of their engineers and bring them into ATG. And that way ATG stays the same size. (Tesler, CHM oral history, p.39)

Who came in mattered as much as who went out.

It tended to be engineers who were burning out. They had five projects in a row that had no space between them and they were long hours and they just wanted a breather. They wanted to try out some ideas they've been thinking of for years and never had time for. And so it was a very strong connection. It was based on Gordon Bell's observation that the only way to transfer technology is to transfer people. (p.39)

The lab had a rule about scope as well: a project had to have a credible path to product in eighteen months to five years, or it did not get started. And it had enemies.

Some of the Product Development managers were idea people and they didn't like the fact that before they even got a chance to come up with their way of doing multimedia we were coming up with a way to do multimedia and sending it to them half built and ready to productize. We thought it was great. [...] But the engineering managers and some of the engineers in Product Development felt like it wasn't fair. They were doing the hard work. We were doing, you know, fun work and we were getting to define the next generation of stuff and it wasn't right. (p.39-40)

In 1990 Tesler moved to the Newton project and handed ATG to David Nagel, one of his direct reports. Engineering management had already attached a condition.

The Engineering Management team basically had made a proviso in there somewhere that we were going to review this policy of this 15 percent and this kind of what they felt was a monopoly of new ideas having to come from ATG. And sure enough, right away that changed. [...] The main thing I remember is that they decided that ATG should have a longer horizon. [...] They wanted to move that 18 months further out and make it be more like three years, five years, you know, and like five to ten years. (p.40)

Then the budget cuts of the early nineties.

Even though they were transferring people out of ATG they weren't having the budget to hire new people and transfer people into ATG. The result of that was that ATG became more and more research-oriented. It was the people who didn't ever want to transfer to product development who were still in research and it was kind of like a university kind of department. We're doing things for the long term. They should turn into papers. They get published. They should inspire people rather than become product templates and Apple could ill afford that in the 90s because [...] the money was no longer rolling in and they were running out of ideas and this group wasn't giving them useful ideas anymore. All the useful ideas had already been transferred and they couldn't come up with new ones because they couldn't hire new people. (p.40)

Early in 1997, with Jobs back as an adviser and Amelio still CEO, Avie Tevanian, then running software, asked Tesler to go back.

"We're shutting down ATG. People say, oh no, we're going to lose a lot of great ideas. Since you started it, would you go over there and find out what the great ideas are that are still there and which ones we should save and which ones we should kill?" So I was sent back there to kill the organization I had founded, but I'm glad he did it. It was the right thing to do. When I got there, I couldn't believe what I was seeing. They were all really cool, but they were irrelevant. (p.40)

His example is a touch-screen conference table with displays under the glass, which he dates as fifteen years ahead of Microsoft's Surface table.

We could sell 10 of these at a time. This is a company that comes out with products to sell in the hundreds of thousands and the millions, not in the tens, wrong company and so I killed that and I killed a few other things, spun out a couple of things. (p.41)

He kept three: a handwriting recogniser for the Newton, which was still alive; QuickTime for Java, cancelled for one day and then reinstated; and a hard-drive search project that became Sherlock, the ancestor of Spotlight. (p.41)

API review with no approver: a mailing list, a one-week clock, and three abuses in fifteen years

Engineering · NeXT and Apple, 1989 to 2011

Bertrand Serlet came to NeXT from Xerox PARC in 1989 to work on AppKit, the framework every NeXTSTEP application was built on, and ended up running all of Apple's software engineering through the Mac OS X years. The thing he brought with him from PARC was a review process for public interfaces.

Why an API needs review at all, in his words:

You can change the code, the implementation, but changing the API, now you need to evolve all the apps that have depended on it. It's very difficult. It takes years, literally. [...] You tell people, I'm going to start deprecating, then you deprecate it several releases. It's typically two, three releases. (Serlet, CHM oral history, p.35)

The usual response is a heavy gate: a committee, an approver, a sign-off. Serlet went the other way.

I was surprised that there was no API review when I came in and the NeXTSTEP API could have been better if they had had a review and that was one thing that we did with OpenStep, we started doing reviews [...] and it's very delicate because you want the owner of the API to own it. You want owners to own, right? You don't want to disempower owners, but you want to have feedback too. (p.34)

The process, in full:

When you have a new API, that's public API, that customers will, developers will use. You have to publish it to a certain forum and you have to address all the feedback, one way or another, you can say, oh yeah, I don't care about that. But you have to address the feedback somehow and it's time-bound. It's a week or some limit like that and after that, that's it. You can go ahead. Your API is approved. (p.34)

The forum was a distribution list.

We had the forum be a distribution list of all the folks that are API savvy and pretty much the trick there is that anyone who has to be on the list is on the list, and that self-determination really works, because people who are not interested in this will not ask to be on the list, right? (p.34)

On abuse:

It's subject to abuse. I think during my tenure at Apple over 10 years, right, 15 years, there were two or three cases of abuse. In those cases, that is the feedback was not addressed or the feedback was this is really a bad API and they went ahead, and you deal one-on-one with the abuse. Very minor. Two or three out of thousands of API reviews. So I'm a big proponent of very light-touch processes, not heavy processes and leaving people ownership. (p.34)

Running the Intel transition with no schedule, only a punch list that had to get shorter

Management · Apple, the switch from PowerPC to Intel, 2005 to 2006

Bertrand Serlet ran Mac OS X engineering when Apple moved every Mac from the PowerPC processor to Intel. A processor change is the largest thing that can happen to an operating system; every piece of software has to be rebuilt, and much of it has to be rewritten. Apple did it in six months, half the time it announced, and Serlet's account is of two management decisions that made that possible: one taken years before, and one about what to tell the team.

The first was Avie Tevanian's, and Serlet enforced it.

One thing that Avie pushed, and I enforced it, is that we always compiled our code for Intel as well as PowerPC, even though we had no Intel machine and so we had some checkers in place in the build system to make sure every single project can be built for Intel and this is years before we did the transition. (Serlet, CHM oral history, p.23-24)

The deal with Intel was signed in February 2005.

We just scrambled to make it happen for the developers conference that was in May, so three months later [...] We wanted also developers to have machines that they can play. But we didn't want the Mac OS to run on any PC. So there was a delicate balance. So we decided to build machines in secrecy and in fact, we enlisted Simon Patience's team, which was the CoreOS team, to actually build the machine that we were going to give for developers at the conference. So for a while, the kernel team was actually building, assembling machines in a secret lab in preparation for WWDC and it was the kernel team, because they were the first one to help make it work. So they all were disclosed. (p.24)

Then the schedule.

We said at that time that it would take about a year to transition and nobody believed it. All the industry thought that it would take much longer. We were hoping it would take less. But we were not sure, because we had a dependency on Intel for some of the new chips coming up. So we were not sure, and still we're not sure, for several months. So I said, well, we're going to pretend we need to ship ASAP and we don't have a schedule, but we just have a punch list of what's left to do and we're going to shrink the punch list and that's what we did. People were upset because they wanted to know the schedule and I said, sorry, I can't tell you the schedule, I don't know the schedule, but it's ASAP, and so we were able to ship in January with the new Intel Macs and so that was six months, not a year. (p.24)

Then two worries.

Now that we were on PCs, people could compare the speed of the Mac on what was essentially an Apple machine, but a PC, so same processor, with Windows NT and so I was very worried by that [...] So I said, let's do benchmarking of lots of things that analysts could benchmark or end users could benchmark. For example, you open 100 Windows, how long does it take, right? That kind of thing and we wrote several hundred tests that we could run both on NT and on the Mac and the results came in, and for 95% of them, the Mac was slower. So we had a big communications meeting. I motivated the team by saying, hey, not so good, and by the end of summer, we had flipped that 95%. (p.24)

The second worry was about the year in which both kinds of Mac would be on sale.

People will find some subtle differences between them, because we've still evolved the Intel code base [...] And that people would play the game of find the difference. So, which I think would be very damaging for moving forward. I wanted the transition to be viewed as invisible. So what we did for the rest of that year is we did software updates that had all the improvements that we made to the Intel side. But that also were all those improvements that were not for PowerPC, but we also included that in the PowerPC update and so we had a single code base. We forced the code base to be the same through software update. So by the time January came in with the new Intel machines, they were exactly the same software as the PowerPC. So no one ever found some bug difference between the two. (p.24-25)

Deciding which APIs to kill by scanning Microsoft Word to see what it would break

Engineering · Apple, the Carbon transition, 1998 to 2000

Nitin Ganatra joined Apple in 1993 answering developer questions in technical support, and by the late nineties he was on the team building Carbon. Carbon was the bridge: Apple had bought NeXT and was replacing the classic Mac OS with a new system built on NeXT's foundations, and every existing Mac application was written against the old programming interfaces. Rewriting for the new ones would have meant shipping an operating system with no Photoshop and no Word on it. Carbon was the subset of the old interfaces that would keep working on the new OS.

Deciding what went into that subset is the story. The instinct was to cut.

We wanted to throw out as much as possible. [...] Because everything that you throw out is just, every API you throw out is an API you don't have to support down the road. Bertrand [Serlet] later, you know, sort of captured that sentiment in a great way by saying that APIs are a liability, you know. They're an asset to your company, but they can also be a liability. The more APIs you have, the more exposure you have to programs that may not work well or make assumptions based on the behavior of those APIs. (Ganatra, CHM oral history part 1, p.28)

The old interfaces were a liability for a specific reason: they had been designed for a machine with 128K of memory, and they showed it.

In the old days, data structures were sort of fully defined and fully fleshed out as part of the definition of an API call. And part of the reason for that was [...] you really don't want to create multiple calls to get and set things out of a data structure. Really, the more resource friendly way of doing that would be to just say, "Here's the data structure, you can get, and pull, you know, get and set things in there all you want." [...] We only have 128K of RAM to run in anyway. So let's just do away with that and just expose the underlying fields and just let the developers do what they want. So fast forward 15 years from, you know, from that attitude, and really, it does become more important to hide the underlying implementation so that you can change it later. (p.29)

Cut too much, though, and the developers would refuse to port.

We had a lot of good data by then where we had these tools where you could scan an application, you could scan a popular application and look and see what are the APIs that it's actually using. And so from that, you know, you could bring up Microsoft Word and run it through these tools and it would say, "Microsoft Word is using 2000 API calls. Here they all are." You know, and you could even flag [...] APIs that we really want to get rid of these ones, but we don't know if we can yet. So let's run through the list of popular apps and see, by getting rid of that API in practice, who are we hurting here? (p.28)

The target Ganatra gives for how far to go:

It was sort of this riding this line between giving developers everything they want and, you know, tying our hands for [...] future development work, or, you know, dial it back as much as we can until developers almost scream that it's too much work to do, but now we're set up well to support them in the future, and just sort of trying to figure out where that was the whole time. (p.29)

Not telling the designers what the hardware could do, on purpose

Management · The iPhone, 2005 to 2007

By the time the iPhone project started, Nitin Ganatra was managing application teams on Mac OS X, and he became the director responsible for the phone's built-in apps. The designers he worked with were Apple's Human Interface team (HI), and the hardware they were designing for was an ARM processor with a fraction of a Mac's memory. The obvious way to run that relationship is to tell the designers the limits up front so they do not waste time on things that cannot be built. Ganatra describes doing the opposite, and why.

Really one of the goals was to sort of let the HI team do, come up with, the best possible design and not bog down the HI team with discussion about number of colors on the display or how much RAM we're going to have or how much, you know, this, that [...] obviously as an engineer you're thinking about those constraints all the time and you're thinking, "Holy crap, am I going to be able to implement [...] this really cool thing that the HI team came up with?" But really you want to, you don't want to start to burden the HI team with these constraints early on. (Ganatra, CHM oral history part 2, p.10)

His reason:

Because then the best thing that they're going to ever come up with is a design that they themselves, the HI team themselves, anticipate will work given all the constraints that were given. You've now put the HI team in a situation where they're now trying to guess at whether their design is still really cool enough to be compelling to a customer but still something that can be implemented. (p.10-11)

It's better to just sort of stand aside and let HI come up with the best possible designs [...] and then we'll figure out if we need to, you know, if we need to implement something that's 90 percent as cool but is possible, on an ARM-embedded device or what have you. We can have that discussion later, but you come up with the best design, the engineering team will come up with the best implementation that we can of that design, and if there isn't [...] a complete match between the design and what can be implemented, by then we'll have a better understanding of what the details are, where it's difficult to implement things, and we'll have, on the engineering side, we'll have specific recommendations for, "Well, yes. You said that this thing could be an unlimited length table, but what if we made it a thousand cells instead?" (p.11)

The same approach shaped the software underneath. Rather than design a grand framework for moving between screens before anyone knew what the screens were, the team built the first apps directly and let shared code accumulate.

If you try to create this architecture too early on, you can kind of stifle the ability to go in different directions or the ability for [...] the HI team to kind of go and go hog wild [...] And so, it's just better to kind of just let it be organic for a while but understand that there's a plan there. (p.10)

That accumulated code became UIKit, which every iPhone app has been built on since.

Three days of memorising a design in one room and redrawing it badly in the next

Management · The iPhone, 2005

The iPhone was built under a secrecy regime in which Steve Jobs personally approved two lists: who was on the project, and who was allowed to see the user interface. The two lists were not the same. Nitin Ganatra, who managed the team writing the phone's applications, was on both. The engineers who reported to him were, for the first few days, only on the first.

For a few days, I had disclosure to see the UI as the manager, but the engineers on my team had not been disclosed yet [...] It was probably for the first three days or four days that we were in this arrangement where Steve Jobs had to personally approve not only everybody who was on the team but then also everybody who had access to the user interface. And so, and I remember I had discussions with Scott Forstall about this too, where I had access to see the designs but my engineers who are actually doing the work here didn't have access. So what should I do? You know, like, should they just go on vacation for three days? That doesn't seem right. (Ganatra, CHM oral history part 1, p.64)

Scott Forstall ran iPhone software. There was a demo due that Friday or the following Monday. The solution was a workaround for a rule neither of them could change.

We had two of my engineers were in one office and in an office right next to them was a computer that had the designs on it. And so, any time they had a question about the designs or some aspect of the design itself or something, what I would do is go into the, and they weren't allowed in the room with the computer that had the designs on it. So I would go into the room with the computer that had the designs on it, play with it for a little bit, try to commit to memory everything that I saw, and then I would come back over and on the whiteboard I would draw, and I'm a terrible whiteboard drawer anyway. And then I would draw an approximation of how things worked and how it behaved and how things should be. And so, we did this for three days. And the whole time we were doing it, we were all perfectly aware of how absurd it was. (p.64)

The interviewer's comparison was to a corporate spy inside his own company. Ganatra's summary was shorter: "It was a little silly for a while there." (p.64)

Antennagate: asking for the complaint data before agreeing to apologise

Marketing · The iPhone 4, 2010

Regis McKenna ran the agency that positioned the Apple II, shaped Apple's marketing from the beginning, and knew Steve Jobs for thirty-five years. By 2010 he had twice declined to come back to Apple, preferring, in his words, to stay friends. So when the iPhone 4 shipped with an antenna that lost signal if you held the phone a certain way, and the story had a name, he was called in as a friend rather than a consultant.

I got a call from Steve, and he was actually in Hawaii with his family. He said he was going to fly back and could I meet him on, I think it was Monday or something like that, and I said sure. He basically called together a meeting of some of the people from Chiat who were still working on his account. He had his marketing people there, and some of the engineering people. And we had a meeting in his conference room, and his son, Reed, was there. He wanted him to observe it. (McKenna, CHM oral history part 5, p.12)

Chiat is Chiat/Day, Apple's advertising agency.

I asked for the data. I wanted to see what the data had to say, because I knew Apple, they had collected a huge amount of data from all the users and all the people that had complained, all those who wanted returns on the phone, and so forth. The iPhone before [the 4] was actually worse than that phone. So it was really a relatively minor issue to the consumers that were buying it, from my standpoint. (p.12)

The previous model had measured worse on the same problem, and nobody had noticed.

I first advised him to just let it go. I said, "It'll fade away, fix it." He didn't want to do that. He said, "No, we've got to address it." I said, "But there are only, really, a relatively few people that are making noise about it. You may be making a bigger issue." We talked about that in the meeting. (p.12)

Jobs overruled him on whether to respond. On how:

Most of the other people were telling him to come out and apologize. They really wanted him to put his tail between his legs and go out there and be very humble and so forth. I said, "Don't do that. I don't think that's you. I don't think that's what you should do. And I don't think the data calls for that, quite frankly. I think you do tell people that technology products are never born perfect, that they will improve, and they're constantly being improved." And he used that line, actually, when he talked. Basically, he said, "We will address the problem." They gave you a little protective band that you put around the iPhone. And, he said, on the next generation, as soon as we can, we'll change it. (p.12)

Once he went out and said these things, they [consumers] didn't lose confidence in the company by saying, "Oh, here's a big failure." The president of Microsoft at the time said this was going to be a failure, this product. And it wasn't. It still continued to sell. The next generation sold even more. (p.12)

Turning the strongest objection into the objector's assignment

Engineering · Apple, the Mac OS X transition, 1997 to 1998

When Apple bought NeXT, Avie Tevanian, who had written the Mach kernel at Carnegie Mellon and then run software at NeXT, became the person deciding which technology survived: Apple's or NeXT's. Every one of those decisions was read inside Apple as a side winning, and this is his account of the one that produced the most noise.

Apple had its own networking system, AppleTalk, which had been designed a decade earlier for small office networks and did one thing better than anything else: you plugged a printer in and it appeared. The rest of the world had standardised on TCP/IP, the protocol the internet runs on, and Apple's own TCP/IP support was bought in from an outside vendor.

The networking standard for Apple at the time was AppleTalk, and AppleTalk was just wonderful networking for a LAN [...] It was way better than TCP/IP. But, the rest of the world has standardized on TCP/IP and it was obvious to me and some others who weren't totally immersed with the Apple way for many years, that TCP/IP was going to win all the battles. It was already winning the battles, and at the time Apple had an AppleTalk team working on AppleTalk, and their IP stack was outsourced, and so I said, "This doesn't make any sense. Networking is a key part of the product." (Tevanian, CHM oral history part 2, p.15)

His proposal was the open-source stack everyone else used, which happened to be the one NeXT already shipped. That was enough to make it an Apple-versus-NeXT fight.

I got no end of complaints about that. [...] In fact, people went around me all the way to Steve saying, "We're going to tank the company if we do this," and including the people who are selling us the stack, the software stack from the outside. They were making money off Apple, and so I just sat down and I laid it out and said, "This is the standard of the future, number one, and number two, this is the standard implementation, number three, it's free and open source. People will keep working on it. This is [a] no-brainer decision. End of discussion." (p.15-16)

Winning with Jobs settled the decision. It did not settle the engineers, and the objection they brought him was a good one.

Good engineers would come to me and they would say, "but you can't do this. Our customers know and love AppleTalk. They just plug in a new computer and it just works and that doesn't work with TCP/IP. It works with AppleTalk," and they're, plug in a printer. It just works. I said, "Great. So your job is to now make TCP/IP do this." (p.16)

Lo and behold, here we are today, 20 years later, using TCP/IP. Everybody just plugs in their computer or their printers and devices and everything and it just works and that's because we got motivated to fix a few problems and fill in a few gaps with TCP/IP that didn't allow it to do that before. (p.16)

He also gave them the arithmetic, and says how it came out.

"What you're advocating is a solution that is going to keep three percent of the addressable market happy." That was the existing customer base. Maybe it's five percent. [...] "I want to keep those people happy but I want to have a way to go get the other 95 percent, and we can't get the other 95 percent with something like AppleTalk. They're just never going to buy it," okay, and, you know, we never got the other 95 percent, but you look today and Apple's doing pretty well even with computers. (p.16)

Not everyone made the switch.

There were some people, they just couldn't get their head around this way of thinking and they either quit or were fired, and then the people that could, they said, "You know what? There's a few things wrong with TCP/IP today, but we know how to fix that, and we can work with the IETF and change the standards and evolve it and do a few things with DHCP and other standards and fill in here and fill in there and make some DNS changes and we can make this all work." And, it does. (p.16)

"Over my dead body," and the two products that argument shaped

Product · AirPort and the iPod, 1999 to 2004

Jon Rubinstein ran Apple's hardware engineering from 1997 and later the iPod division. Two of his stories are one story: an argument he lost about Wi-Fi, and how losing it changed the way he fought the same argument about the iPod three years later.

AirPort was Apple's 1999 Wi-Fi product, a base station and a card, at a time when Wi-Fi was one of two competing standards. Rubinstein had spent time in Washington keeping the other one, Intel's HomeRF, from getting the rules changed in its favour, and when Wi-Fi won he saw what Apple had.

Wi-Fi became sort of the de facto standard and I'm looking at this and I'm going, "This is a big business." Right? Our base stations could be used because we're just a PCI card, right, so we could take our PCI card that goes in our Mac, plug it into a PC. We could take our base station. We're missing one thing, and that's the application that runs on a PC to configure the base station. It's not a big deal. (Rubinstein, CHM oral history part 1, p.81)

He took that to Jobs.

So I go to Steve, and I said, "Look, I need three, four people. I'm going to port our custom app on the Mac over to a PC version, so we can enter the base station business on the PC." Steve goes, "Over my dead body." I said, "But Steve, it's like three, four people. I mean, we already have it running. We just need to productize." "No." All right. So I'm like, "All right." "No." I mean, I had lots of other things to do, so... (p.81)

Jobs's reason was that AirPort was part of what made a Mac worth buying. Rubinstein's counter was that Apple's base stations were better than anything on the PC side, with features nobody else had.

We had mesh. No one else had mesh. Until Eero came out recently, no one else really did mesh. We had that. We had easy configuration. There was no SSID, right? I mean, it was the name of a network. I mean, it was Mac-like, right? Everything was simple. [...] I said, "This is going to be a multibillion, I want to build a multibillion-dollar networking business." Right? And Steve went, "No way. We're not doing that." I said, "All right, so we're not doing that." So we kind of put that aside, right, and, which then came when we did the iPod later on. I dug my heels in. (p.81)

The iPod shipped in 2001, Mac-only. To get the music labels to license songs for the iTunes Store, Jobs told them the Mac's market share made it a safe experiment.

Steve convinced the music companies that, "Look, this is a great experiment because at most we've got 2% market share. At most we've got 2% market share, so you do not have to worry about 98% of your market, and so it's a great way to experiment," and so they went "Okay." And frankly at the time we were going to stay Mac only. We didn't have this idea we were going to expand to the PC. (p.92)

Then the same argument as AirPort, with a different outcome.

It makes people buy more Macs, and that was the whole point of the iPod, was to sell more Macs, but the lesson from AirPort just sort of sat on my shoulders and sat on Phil Schiller's shoulders, and we started hammering Steve. "We got to take this stuff to the PC. We got to take it to the PC," and we just kept haranguing him, and finally he goes, "Fuck you guys. I don't want to talk about this anymore. You do whatever you want." (p.93)

What they did with it:

So we went out, and we found a software company. I think they were in L.A. [...] We basically hired them to take their PC software and hook it up to the iPod. Music Match is what it was called. [...] They'd done a product that was sort of a half-assed version of iTunes, and it wasn't very good, but it worked on a PC, and it worked with portable players, and so we just modified it to work with the iPod, and then we gave them a very strict roadmap, and we said, "Look, here's the roadmap we want, and this thing's going to put you guys on the map, so here's what we need." And Phil and I managed them very closely and managed their roadmap and the iPod on the PC. (p.93)

The same passage carries a second reversal.

By the way, we also did the Mini at this point in time, which Steve tried to cancel. I got a panicked phone call one day. "You better get to this meeting. Steve just cancelled the Mini." And I'm like, "Ahhhhhh," so I come running down there, and Steve's just leaving the room, and everyone goes, "What do we do?" I said, "Just keep going." Because the Mini is really the product that caused the iPod to take off because of the price point, and it was the anodized aluminum, the colors, [the] price point. It had enough storage but not too much. (p.93)

The iPhone developer story that lasted from a Friday to a Monday

Product · The iPhone SDK, 2007 to 2008

In June 2007, weeks before the first iPhone shipped, Steve Jobs told developers at Apple's annual conference that they could write applications for it using web technology, running in Safari. No native software; the phone's own apps would stay Apple's. By October Apple had announced a native SDK instead. Four people who were inside that reversal describe it, and they do not fully agree.

Richard Williamson ran the iPhone's Safari and WebKit engineering, so he was the person best placed to argue for the web approach, and had already stopped believing in it.

I and the iPhone Safari team had a pretty deep background in WebKit and web technologies. And so, I was in a position to try and advocate for that, although I had already come around to the idea that [...] native development was far more efficient than web technologies because of the tools, because of the fragility of web technologies, and [...] if you look at the Objective-C frameworks, they kind of guide you towards good development, guide you towards how to build an app. And HTML was never designed to do that. (Williamson, CHM oral history part 2, p.27)

On where the web-apps announcement came from:

Within the engineering organization, there wasn't a final approach to say, this is what we're going to do for developers. In fact, it was only a few days before the announcement that Steve made [...] that we heard that there was a great developer story. And really Scott [Forstall] and Steve huddled and came up with this. It wasn't vetted by me, certainly, and I don't think Nitin or Henri. So, we had to really scramble to try and figure out what that meant. And I think it was pretty clear to the engineering team that it wasn't going to be a great solution. (p.27)

Scott Forstall ran iPhone software; Nitin Ganatra and Henri Lamiraux ran its applications. Williamson's read on why it happened:

I think Steve said to Scott, "Look, we've got to do something about third party developers. What are we going to do?" And it was one of these snap things that Steve just wanted something and had to have something. And Scott came up with the story. (p.27-28)

Ken Kocienda, who built the iPhone's keyboard, remembers how long the story lasted inside the building.

I do recall there was like a Friday afternoon where some Safari and WebKit people from the outside team came to our iPhone hallway and were looking around at offices. "Oh, I'm going to sit here. I'm going to sit here," because they were actually going to come and do this web development story. And that was like Friday. And then Monday, it was like redone. It was like a very quick turnaround that we're going to do this story. We're going to do this story. We're going to do this story. We're not going to do this story. (Kocienda, p.28)

Ganatra was not in the room for the decision, but he describes the argument that did the work, and it was not abstract. They took one real application and tried to build it the way they were telling developers to.

Our very first, you know, our first target was Epocrates. And we were kind of thinking, "Well, what if there was a doctor and they're walking around with a Palm [...] Treo [...] and they were running Epocrates? Well, what would they do?" And so, there was some internal work around just kind of going through the exercise. Well, what would it mean for Epocrates to actually ship a web app? And I think it became, very quickly we realized [...] they would have to do an enormous amount of work [...] and by the way, once they had done all of that [...] what they would end up with, was an app that behaved nothing like any other app on the system. It would probably take longer to launch. It probably wouldn't have the nice smooth scrolling. (Ganatra, CHM oral history part 2, p.43-44)

Epocrates was a drug reference that doctors ran on Palm handhelds, exactly the kind of customer the iPhone needed to take from Palm. Ganatra's summary of the exercise:

"Well, what if we did want to go and target this client, and wanted them to make a really great web app, what would they have to do?" Answer: "An enormous amount of work, and they'd end up with something not great anyway." So, I mean, you know, I think we went through enough examples like that until we realized that really the right answer here is to release an SDK. (p.44)

The general principle he draws is about having two technologies.

If you've created two technologies, one for your own use, and one for third parties, or for somebody else to use, then that must have been for a reason. It must be that either the things that you're doing are so different from what you expect third parties to do that you need to have some brand new [...] technology that's different, or you think that what a third party will be doing or could do is not as important or as compelling or interesting as what you may be doing on your own. So that's why you have two. In either case, it's not a great answer. (p.43)

What made the reversal affordable was something Forstall knew about his own organisation.

Scott Forstall, to his credit, he knew what we were, you know, what these projects looked like and how we were already building the software. He had a good understanding that, internally, even though we didn't have a third-party SDK, internally, we were developing these things as though we had, you know, using our own internal SDK. So then the amount of work that we would have to do, we already have an SDK, basically. So, we would have to do some sanitizing and cleaning up and getting some interfaces ready to share with the outside world. [...] You're going to be happy with those interfaces for ten or fifteen years or twenty years. (p.44)

Bertrand Serlet, who ran Mac OS X and argued for the web side, adds the precedent.

A few years prior to that, we had on the Mac, a feature called dashboard widgets. That was very, very small, lightweight apps [...] We came out with dashboard widgets and within a couple of months, we get tens of thousands of dashboard widgets and so we knew that there was pent-up demand for lightweight applications on the platform. [...] So we argued how to open up, web app versus Cocoa. We did Cocoa, and we opened up and a year after, we started getting apps and I thought we'd get thousands of apps based on this dashboard widget experience, within a few months. I was totally wrong. We got 100,000 apps. (Serlet, CHM oral history, p.29)

Serlet also notes that Jobs "kind of liked the fact that all the apps were Apple apps" and "was actually arguing for a while that maybe we should keep it closed." (p.29)

The owners of the framework argued against using their own framework

Engineering · The iPhone, 2005

When the iPhone project started, the obvious way to build its software was to take AppKit, the framework every Mac application's interface is built on, and put it on the phone. AppKit was mature, the engineers knew it, and it embodied the Cocoa programming model that had come from NeXT and that Apple had spent a decade getting developers onto. Nitin Ganatra, who was about to manage the phone's application teams, describes the meeting where that option died, and who killed it.

The first option was to actually bring AppKit itself, which was the key component of Cocoa and make that the phone. (Hsu, interviewer, CHM oral history part 1, p.57)

The people making the case against were Ali Ozer, who managed the AppKit team, and Kristin Forster, an engineer on it. Ganatra's account is as much about engineers in general as about them.

Engineers are very sort of proud of the work they've done and they want to, we want to, believe that the things that we've created are, can be suited or can be used for anything, including things that we never anticipated they be used for. And a lot of times that's just not true, you know. But I think, some amount of the pride of ownership and pride of development kind of clouds your judgment in some ways and makes it so that you, you might think that the thing that you've developed is far more capable than it actually is. But that's the, but to Ali's credit, that's not Ali. And that's not Kristin. (Ganatra, p.58)

What they knew that nobody else in the room could have:

They were well aware of all the ways that the event system and that menus and that mice and that, you know, all these things that were very essential to how desktop computers work, they were so embedded into how AppKit handled events and did so much of what it does that they just, they themselves, Kristin and Ali, were making the argument that it's just not well suited for something like touch or for multiple layers on the screen or things like that. (p.58)

And why that settled it:

The fact that they were arguing against using AppKit to me was the strongest argument that okay, I mean, these are, they're super smart people and they are, and they understand their technology better than anyone. And so why would we try to use AppKit when the experts are saying we really shouldn't. (p.58)

Ganatra adds that Forster "did the majority of the research as well." (p.58)

The alternative was to keep the Cocoa programming model and write a new framework for touch, which became UIKit. Every iPhone application since has been built on it. The decision to build it rather than adapt what existed was made by the two people with the most to lose from that answer, on the strength of research one of them had done into her own code.

Choosing a browser engine: 150,000 lines you can hold in your head over 1.5 million you cannot

Engineering · Safari and WebKit, 2001 to 2002

In 2001 the Mac's web browser was Internet Explorer, which belonged to Microsoft, and Apple wanted one it controlled. Don Melton, who had managed part of the Mozilla project at Netscape, was hired to build it, with Ken Kocienda as his first engineer. Neither had built a web engine. Then Richard Williamson arrived, back from a year travelling, having asked Bertrand Serlet if there was anything interesting to work on and been offered two things.

He told me about two things. The browser project, and he said, "It's just starting. We're trying to figure out what to do." And then the other was, there was an effort that was super-secret at the time to go from PowerPC to Intel. [...] Of those two things the browser sounded far more interesting than working on the Intel project. So I said sure. And I didn't know anything about how to actually build a Web engine or a browser. (Williamson, CHM oral history part 1, p.24)

The question was never whether to write an engine from scratch; that fell away fast. It was which existing one to start from. Kocienda describes the state of the decision when Williamson joined.

Don and I were floundering around for a little bit of a while for a few weeks before Richard came on board. And Richard very, very quickly said, "Now, what are you guys doing?" [...] We were looking at Mozilla. We were looking at maybe licensing from Microsoft, Internet Explorer, for Opera versus iCab versus, there's KDE and GNOME, these Linux desktops or whatever. And Don and I were just kind of tripping over ourselves. We didn't really know what to do. Too many options. And so Richard goes away and just says, "How about KHTML?" (Kocienda, p.25)

KHTML was the engine inside Konqueror, the browser of the KDE desktop for Linux. Almost nobody outside that world had heard of it. Williamson went away for two days.

He calls us in for a demo and says, "Hey, guys look." And he's got KHTML working on a Mac. [...] He just convinced this KHTML code that yeah, yeah, yeah, you're running on a Linux machine even though it's a Mac. No, its X Windows. It's fine. Don't worry about anything else. And we were floored. (Kocienda, p.25)

Then the arithmetic.

Mozilla was the leading candidate at that point, the other leading candidate but it was a million-and-a-half lines of code. And KHTML was 150,000 lines of code. [...] It was three guys and we figured 50,000 lines of code each. (Kocienda, p.25)

Williamson had looked at Mozilla's code.

Beast. I think an important thing in software development is that the software does what it needs to do well and doesn't do things that it doesn't need to do. And the Mozilla code base has so much stuff in there that's irrelevant for a Web engine. Interesting technology in other domains but really irrelevant in terms of building a Web engine. And in order to get your head around a piece of software you need to be able to context switch it into your head and really understand how it works. It's almost impossible with Mozilla because of all of these irrelevant subsystems. (Williamson, p.27)

Kocienda had tried the other path and has the numbers from it.

Mozilla source code base had 20,000 typedefs. So you want to know what is in that software, you've got to learn 20,000 names. [...] I actually did get Mozilla running on Mac OS X which was a weeklong, which was a horrible, horrible trial because Mac OS X was so new that Mozilla didn't have a port. And this was secret. So you couldn't ask anybody. So I had to figure it all out for myself. So I got Mozilla running on Mac OS X but it basically never, I don't think it ever rendered a webpage. It would just crash. [...] So after a crash I'd look at the core file. It's 100 levels deep, 100 frames, 100 stack frames deep. So how are you going to context switch this into your mind? (Kocienda, p.28)

Opera was a serious option too, more advanced than KHTML, and they had a trip scheduled to go and talk about licensing it. They never took the trip. And Melton, who had managed Mozilla, "knew Mozilla too well to want to work with it again." (Kocienda, p.26)

The decision still had to get past Avie Tevanian, who ran software. Melton set one condition for the pitch.

Don said, "Okay, we're going to do the presentation but we're not going to use PowerPoint or any other application. We're going to do the slides in this application. And then we'll project that." So that's what we did. We actually made HTML slides and presented it in the browser and it was flawless. And we didn't tell Avie until the end of the presentation [that] we had been using the application. (Williamson, p.28)

One more thing was decided at the start.

We were hired to make a browser app that we could go and replace Internet Explorer as the double clickable app in the Dock. But we also from the very, very beginning we needed to make a framework. We needed to make a developer toolkit. That was part of the plan from the very, very beginning. And so WebKit was not after thought. (Kocienda, p.27)

Asking the customer to write a letter so he could win an argument with his own investors

Management · Stepstone, mid-1980s

Objective-C did not start at NeXT. It was created at a company called Stepstone, and Steve Naroff joined Stepstone to work on its compiler in the mid-eighties, before NeXT was a customer of any size. The compiler he found was, in his description, a translator: it turned Objective-C into C without understanding much of it.

It was naive. It wasn't full-bodied. It was not capable of detecting programming errors that are common, even a misspelling. Part of the reason it wasn't detecting errors was not only the technology but the language definition. There was no explicit interface declaration whatsoever. So if you typed a message expression and sent an object a message of a certain name and typed it incorrectly it would just assume that that was the name of the method. (Naroff, CHM oral history part 1, p.18)

Misspell a method and the compiler would agree with you. Naroff wrote a proposal to fix the basics: interface declarations so errors could be caught, a compiler that processed the whole language, and generated code that worked with the standard Unix build tools instead of silently producing broken executables. He could not get it funded.

I fought pretty hard for fixing the basics, which, to me, it was as obvious as [...] yet I got pushback from the venture capitalists who were thinking in dollar signs, thinking, "Why should we pay you to recraft and add some of these things that will make it more robust?" To them, error-handling, they're money people. This notion of, "Well, we can't flag an error," to them, that's like, "Well, so what?" They're not programmers. So I was getting a little bit frustrated when I was basically being somewhat ignored. There was some appreciation but this person, Ken, who was leading Software, he didn't have the technical chops to, as the guy who was running Software, to fight for me. I had to fight myself. (p.19)

What changed it was a visit from the customer.

The great fortune was NeXT arranged a trip for two of their engineers, Steve Stone, who was an OS guy, and Trey Matteson, who was just out of Brown, really bright young guy. They both came to visit us in Sandy Hook and read my proposal and were just jazzed, totally jazzed. They're, like, "Oh, my God. This is exactly what we're bumping into. This solves our problems." (p.19)

He asked for something he could carry into the next meeting with his investors.

I said, "Guys, when you go back to Palo Alto I need you to write a letter that identifies what we talked about and why you were supportive of this, so I can basically do the political dance at Stepstone." And they wrote the letter and I have the letter to this day. I think you've seen it. After that they had to let me work on this stuff. So that was how I got to add interfaces and fix some of the runtime problems. (p.19)

The interface declarations he got to add are still in every Objective-C header file today.

Objective-C++ in two weekends, on top of a stranger's open-source work

Engineering · NeXT, around 1990

NeXT's frameworks were written in Objective-C. The companies NeXT most needed as developers, Lotus, Adobe, Pixar, had large codebases in C++. The two languages could not be mixed in one file, so anyone who wanted to use NeXT's frameworks from C++ code had to bridge between them by hand. Steve Naroff ran NeXT's compiler work, and this is how he solved it in his spare time.

The compiler NeXT used was GCC, the free compiler from Richard Stallman's GNU project, which handled C. Naroff had already extended it for Objective-C. The C++ front end for GCC was being written by someone he barely knew.

I briefly touched base with Michael, to get the lowdown on the state of the compiler, and he was pretty optimistic that, though it wasn't finished, it was compiling quite a bit of stuff from the C++ perspective. So, because he gave me enough positive feedback about his work, and, again, I didn't really know Michael, but he seemed really smart, and I trusted him, the little bit I knew him, I decided to try, on a weekend hack, to take his work and add my Objective-C work to it. (Naroff, CHM oral history part 1, p.37)

Michael is Michael Tiemann, who went on to co-found Cygnus, the first company built on free software. The two sets of compiler changes had to be reconciled, because GCC had not been built to be extended by two people at once.

I got Michael's work in a weekend hack, got it pretty far, and was really pretty blown away that, I mean, the grammar was complaining about this, that or the other thing [...] But I basically didn't let it bother me, and I just went heads down and continued. Long story short, after a couple weekend hack fests, because I didn't have time during my normal hours at NeXT to work on C++, got it far enough that I shipped [...] Lotus the compiler, and they were like giddy that, "oh my god, this is like doing a lot of good for us. And please finish this." And that work ended up being used by them, and then the Photoshop team, and Pixar, and it became really popular. (p.37)

What made it possible in two weekends was a design decision about what not to attempt. The ambitious version would have merged the two object models, letting an Objective-C class inherit from a C++ class.

What I wasn't doing, which some people initially thought could be the design point, is to merge the object models, right? Like allow a C++ class to coexist with an Objective-C class. For example, you know, that would allow potentially sub-classing in Objective-C, a C++ class. I thought that was just idiotic, and even if someone was lobbying hard for it, I would just say, "Listen. That's not the design point. It just doesn't make sense." [...] That would be a research project. (p.37)

He kept the two languages separate and let them sit in the same file. Lotus sent him a t-shirt, with a letter he still has.

"As a token of our appreciation for your efforts developing Objective-C++, been authorized to present you on behalf of the Lotus Back Bay team, the coveted code talks, bullshit walks t-shirt." Okay? "The Objective-C++ compiler correctly compiles all our code, allows one to use all the features of both Objective-C and C++ in the same file, and was finished and delivered before we expected it." (p.38)

Naroff on why it worked:

I wish I could say, in retrospect, we were all so brilliant that, you know, all this great stuff worked, because we're such great planners. There was so much serendipity here. We didn't control Michael Tiemann. He came out of the woodwork. And doing what he did is serious work. C++ is a serious language, and it takes a very special person to do what he did. And we leveraged it. (p.38)

Putting Objective-C in the kernel so ten engineers could cover every PC on the market

Engineering · NeXT and Apple, 1993 to 1997

In 1993 NeXT stopped making computers and became a software company selling NeXTSTEP for Intel PCs. That meant supporting hardware NeXT did not control: every network card, every disk controller, from every vendor, each one needing its own driver. Blaine Garst was a NeXT kernel and runtime engineer.

The economics were bad before a line of code was written.

We tried our damnedest to build a profitable product. And at some point the hardware, we couldn't make it on Motorola hardware. We shifted to Intel, tried to be an Intel-based NeXT computer box, but we had to pay the Microsoft tax. Every piece of hardware we sold had to pay. We had to pay 60 bucks to Microsoft because the manufacturers had this anti-competitive [arrangement]. They later were found to be guilty of monopoly practices in this manner. (Garst, CHM oral history, 2:46:59)

Then the driver problem, and the fix.

In order to be viable on an Intel we had to deal with a bazillion different drivers out there. There's a huge space on Intel. And so I put Objective-C into the kernel so that the kernel team could just subclass a new Ethernet driver. Just tweak, you know, with inheritance and just tweak a little bit and new Ethernet driver done. You know, and so that's how a team of 10 kernel engineers could put out a NeXTSTEP that ran on Intel, on many Intels. (2:47:39)

Most network cards from a given family differ from their siblings in a handful of registers. In a language with inheritance, a new driver is the old driver with those differences overridden. Without it, each one is a copy with edits.

Then Apple bought NeXT, and the kernel came with it.

They later shifted that Objective-C driver interface over to C++ when they got to Apple, right? I still, the kernel team still won't tell me who exactly did it. I think I know who did it, but because Objective-C was unknown to anybody at Apple and they just feared it and they wanted C++. So they made a C++ interface, but they change something in the compiler every release. And so now they're using an outdated compiler because it's the only compiler that'll generate [it]. (2:48:13)

The interface that replaced his is the one Mac drivers are still written against.

Prototyping a feature, getting his boss excited about it, then arguing it down himself

Engineering · The NeXT-to-Apple transition, 1996 to 1997

Objective-C sends a message to an object with square brackets: [object doThing]. Every other mainstream language uses a dot: object.doThing(). For developers arriving from C++ or Java the brackets were the first thing they hit and the first thing they complained about, and in the months between Apple announcing it would buy NeXT and the deal closing, Steve Naroff, who ran NeXT's compiler work, built the obvious fix.

It sort of made sense as the new Apple that we might want to start fresh with this new syntax. Because it's more akin to C++ and Java. And I am, I'm not 100 percent certain that I prototyped it in isolation and then showed it to people, or whether someone sort of said, "Why don't you prototype this?" [...] But so I did it, and Avie got excited. And Ali Ozer's [AppKit] team was not thrilled. So perfect example of the tension between, wow, external programmers, that have never seen Smalltalk, aren't familiar with NeXT, but just want standard dot notation. They'll like this work. But the traditional NeXT people are having a fit. (Naroff, CHM oral history part 1, p.65)

Avie Tevanian was his boss and about to run all of Apple's software. Ali Ozer managed AppKit, the framework the syntax would be used against. The people who wanted the change were the developers Apple needed to attract; the people who hated it were the ones who had built everything so far.

Why I ended up being against it was, again, much more pragmatic. I said, "Oh, my god! Objective-C++ has been such a big thing in NeXT world, it's going to be probably just as necessary in the Apple world." And the beauty of Objective-C++ is the Objective-C message expressions are clearly segregated and distinct from the C++ code. [...] If we make Objective-C message sends look like C++, you will not be able to tell the difference. (p.65)

Objective-C++ was his own earlier work: the ability to put both languages in one file. It worked because you could see at a glance which line was which. Dot syntax would erase that.

Now, you could argue, "Oh, well, we're going to have a fancy code sense editor, and you know, the code editor will know, which is true, there's no doubt. The code editor can know, and the language knows. So there's no ambiguity in terms of the language, but just at a superficial level it makes the two languages more integrated, which is why people were excited about it on one level. So I thought keeping them semantically, syntactically and semantically distinct was better than syntactically similar, but semantically distinct. (p.65)

In the end, I sided with Ali's guys and said, "I can't in good faith push this." And I think Avie was very disappointed. And if my only goal in life was to make my manager happy, I would have given in. I would have, and he would have made the resources available. [...] But it's funny, because it just sort of dissipated. It just, and I used to have documents that, for all this stuff. And even that, I abolished from my library. There's no remnant that I could find of any of that. (p.65)

A decade later Apple added dot syntax to Objective-C anyway, for property access, and Naroff says on the same page that he still thinks he was right.

Hiring off the code instead of the resume, then setting the task that made a new compiler arguable

Management · Apple's developer tools, 2005

For twenty years Apple's compiler was GCC, the free compiler from the GNU project. It worked, and it was structured, on purpose, to resist being used as a library: its authors did not want proprietary tools built on top of it. That made everything downstream harder than it needed to be. An editor that understands your code, refactoring, fast incremental builds, good error messages: each one needs a compiler you can call as a component, and GCC would not be one. Steve Naroff, by then running Apple's compiler group, had lived with this since NeXT.

In 2005 an engineer in his group pointed him at a graduate student.

When I looked at Chris's background and what he was doing, I was blown away because, to be honest, I hadn't hired many PhD students because most of them didn't have the hands on the keyboard as much as hands on writing papers. Not to take anything away from people who write papers, but Apple was much more oriented for people that had big ideas with writing code. And Chris's LLVM was open source as well. And so, I was able to look at it. It was much better than looking at the resume. (Naroff, CHM oral history part 2, p.14)

Chris Lattner had written LLVM, a compiler back end designed from the start as a set of libraries, as his PhD work at Illinois. Because it was open, Naroff could read the thing itself rather than a description of it.

Chris had went even further to, I'd say, make himself attractive to me and to a company like Apple where he integrated his LLVM work as a backend to GCC. So, it was like just amazing work on so many levels. And we brought Chris in. And Chris, it was just apparent from the first fifteen minutes that this guy is an amazing developer, designer, person. It didn't take more than fifteen minutes for me to realize we've got to hire this guy. (p.14)

Then what he gave Lattner to do next.

The other non-trivial goal I put on his task is to actually compile the entire system with GCC as the frontend using his LLVM backend, which I don't know how long it took him, but it far exceeded what I thought it would take a mere mortal to do. (p.14)

After he did that, since I'm a frontend guy [...] I said, "You know, Chris, it would be great if we can finally make a compiler, a full frontend, middle, backend, that truly is library-based that is truly going to meet our compile time goals. And while GCC has been great for well over a decade, it's time for a new compiler that will support the IDE." [...] So, I pitched starting Clang. (p.14)

Clang is the compiler every Apple platform has been built with since, and LLVM is now underneath much of the industry's tooling.

"You're not good enough": told he could not present his own work, and calling the car to ask why

Leadership · Apple, 2002 to 2003

Steve Naroff ran Apple's developer tools. In 2002 he took four engineers and some of Jobs's interface designers and spent four months prototyping a redesign of Project Builder, the tool developers wrote Mac software in, so that it looked and behaved like the rest of Mac OS X instead of like the NeXT application it had been. It became Xcode. This is about the meeting where he showed it.

We arranged a meeting with Steve Jobs and Phil Schiller and Ted, and I forget exactly who else was there, but those were the main, Avie was there. That's right. So we had lots of vice presidents in the room, and I gave Steve the demo, and Steve was thrilled, absolutely thrilled. What I thought was going to be a 15-minute demo ended up being a lot longer, because there was a lot of talk in the room. [...] Steve had said, "Well, we have to demo this at my keynote at the developer conference," which was great news. I was like, wow, that's great, and he said, "Who should demo it?" and since I had just given a pretty good demo, I was feeling great. I said, "Well, I'll do it," and he said pretty quickly, and I'm not so sure these are the exact words, but, "Oh no, you're not good enough." (Naroff, CHM oral history part 2, p.4)

I was like, because it's a room of VPs. I had just done this great thing, and he's happy. Well, that was terse, "I'm not good enough." So I shut my mouth, because it wasn't worth getting into it with him there, and he said, "Let's get Chris Espinosa," who was, as you know, with him in the garage, still at Apple today, and Chris worked for me I think at the time working on AppleScript. But Chris was a great presenter, and, yeah, he's very charismatic on stage and just a great speaker. So I said, yeah, cool, so let's ask Chris to do it. Okay. I was quickly past the hurt feeling. (p.4-5)

He was not quickly past it.

The meeting ended. I went back to my office, sulked a little bit like I usually do when that type of thing happens, and I decided I'm going to pick up the phone and call Steve. So I called him, and his, he had someone that was a dispatcher that would find him, so she picked up, and she said, "I'll find him, no problem." So she patched me through. He was driving, and I said, "Steve, why'd you have to do that? I mean, what was that about? You were happy. I just worked my butt off," and he said, "Listen, Steve. I trust you with engineering. You're a great engineer. You've been a great manager over the years. You're not a great presenter," and he said, "It's my stage, and I need to make sure I have the best person there. You've done great stuff. I'm not taking away from that." (p.5)

I said, "Well, that's great to hear, Steve, and I agree with you. I just wish you would've said it like that in the meeting," and he just said, "Well, they all know you're great, and so I didn't have to say it." (p.5)

Naroff, years later:

Looking back, there's no doubt that sometimes I'm sure I was a little thin-skinned, but when you're in the bomb run, the trenches, whatever you call it, and you're working really hard, it's tough to be talked to like that. But what was great about Steve and why we always maintained a great relationship [...] is he understood and, in fact, then obviously patted me on the back and made me feel good. So he was someone that respected that I picked up the phone rather than let it linger or hold a grudge. (p.5)

Chris Espinosa gave the demo at WWDC 2003.

The bug thirty engineers could not find, and the phone that shipped with it

Engineering · The iPhone, 2006 to 2007

A phone has two brains. The application processor runs the operating system and everything you touch. The baseband is a separate processor with its own operating system, running the cellular radio, and in 2007 Apple bought it as a black box from a vendor. The two talk over a serial line. Andy Grignon ran the iPhone's radio engineering, and his account starts with how many other things were new at the same time.

Now we've got everything changing. The apps are different. They have to be custom. We've thrown in fingers as the input mechanism, instead of a mouse, which was actually a pretty big difference. We had never built a keyboard before. [...] So, we had every layer of the stack custom, different, new, full of bugs. And it should come as no surprise that, by the way, with a deadline that was immutable, right? This deadline couldn't change. And Steve was betting the company on it. (Grignon, CHM oral history, p.22)

New chip, designed in-house. Operating system ported to a processor architecture it had never run on. New toolchain. New apps, after the plan to reuse the Mac's had failed. What that does to debugging:

Imagine you're an engineer in this morass, right? And you go to Mail, you check the thing and the phone reboots, which happened all the time. Whose bug is it? Is it the operating system? Is it Mail? Is it the tool chain? Or hey, maybe it's the bug in the silicon, which happened. (p.22)

Several times the whole program stopped because nobody could get past a problem, and the people who had written the software that generated the chip had to be flown in to sit with Apple's hardware engineers. Grignon's explanation for why vendors agreed to that is that they thought they were working on the next iPod. Then the worst one.

One of the worst ones was we had, of all things a problem with what's called a UART, and the UART is the serial block of a chip. It's been around since the dawn of chips. It's the oldest thing ever. It's a serial line, and ours had a problem with it so that the connection between the chip that made phone calls and the main chip that ran all the software in very certain circumstances but not as rare as you'd think, that link would go dead and that would result in effectively a dropped call with the chip that made the phone calls rebooting itself, and so, you now couldn't make a phone call until it came back up. (p.23)

The signal bars went to zero every time it happened. After the phone shipped, Grignon changed his license plate to ZROBARS.

We couldn't figure it out, and you'd have a guaranteed reliable link between these two chips and it's failing. Why? We had rooms full of people, like 30 people who would single step through to see what the hardware is doing. Register by register, it latches this, it does that. Nobody could figure it out. (p.23)

They shipped anyway. The fix was not a fix.

We actually shipped the very first phone with this bug in it, but [...] we took a piece of the Bluetooth stack [...] HCI, something like that. It was a piece of the Bluetooth stack, 'cause Bluetooth radios give you a serial port but it's over a lossy connection, so it's effectively a UART over wireless, which, so Bluetooth solved that problem of a lossy UART, which is supposed to be a hardwired thing. So, we implemented that between our main processor and our phone processor so that when the UART shit the bed that layer kicked in and it kept the link alive and it effectively did a reboot but the chip didn't restart itself. (p.23)

Bluetooth has a layer whose whole job is to keep a serial conversation going over a radio link that drops packets. They took that layer and put it on a wire that was never supposed to drop anything.

Later in the interview Grignon comes back to why it was missed, and the answer is about testing rather than hardware. He had written the bring-up software for the chip, the code that runs before any library exists and asks whether the silicon can add.

You'd be surprised at the things you catch, and we should have caught things in that UART that went sideways. Like, we wrote a bunch of unit tests that test, can this thing move data? Yes, it can move data. We didn't check all of the conditions, and so, that was where we got bit on that. (p.46)

Shooting a process in the head instead of auditing every allocation

Engineering · The iPhone, 2006

The iPhone's software was built from Mac OS X, and Mac OS X assumes it will never run out of memory. On a desktop that is true enough: when physical memory fills, the system pages to disk and slows down. The phone had no paging and a fraction of the memory, so running out was not slow, it was fatal. Richard Williamson, who led the iPhone's Safari and WebKit work, describes what that meant for code inherited from the Mac.

It's pretty clear that we had to do something because the frameworks on OS X are written with the assumption that you have unlimited memory because you have virtual memory. So if you run out of physical memory you swap. And so almost no software is written with malloc, assuming... (Williamson, CHM oral history part 1, p.72)

Ken Kocienda finishes the sentence: "Malloc can't fail." Malloc is the routine every piece of software calls to get memory. On the Mac it always succeeds, so nobody checks whether it did.

Nobody checks whether or not they get a pointer back, whether they got a return value. So we weren't going to rewrite a lot of the software that we imported from OS X onto the iPhone. So it's pretty clear early on we had to do something. (p.72)

The thorough fix was to go through every allocation in every framework and add the check. Nobody had time for that. Williamson's alternative:

The solution was, use a gun and shoot a process in the head if it's misbehaved. So the idea was, in the kernel we were monitoring memory allocations. And if memory levels were getting low or there was an application that was abusing memory we'd just kill it and give precedence to the foremost application. So when you dismiss an application on the phone it doesn't necessary exit. It's given a bunch of opportunities to do things before it has to exit. And if it doesn't exit then we kill it with Jetsam. (p.72)

The name is the nautical term for cargo thrown overboard to save the ship. Rather than make every application handle scarcity, the kernel watches the total and throws applications out, starting with the ones you are not looking at. The application never learns memory was short; it just stops existing.

We met with the kernel guys and pushed this idea forward. And they were like, you know, you're crazy. This is ridiculous. What's going to happen? How are things going to get cleaned up? But eventually they came around. And we implemented it. So it's remarkably seamless, how memory management works on the iPhone. You know, you really don't notice when applications exit and start. (p.72)

Jetsam is still what does this on every iPhone. It is why an app you left an hour ago sometimes reopens from scratch, and why the phone almost never tells you it is out of memory.

The same passage records an argument Williamson lost. He wanted more RAM.

I was a huge advocate for more RAM and the hardware guys went no. And we're talking about pennies. But they insisted on living with the RAM that we had. (p.72)

iMovie's interface shipped as a live Photoshop file, and nobody was told for four versions

Engineering · iMovie, 1999 to 2000

In 1999 there was no pipeline from a design file to a running interface. A designer produced an image; an engineer cut it into pieces, gave each piece a name, and compiled the pieces in. Every visual change, every shade of a button, cost a build. Glenn Reid, who had come from NeXT and was running the small team building iMovie for the iMac, describes what he and a designer did about that.

iMovie, no one knows this except now everyone knows this because I'm being recorded, but we had the same problem. Like we were writing a C program to make movies. We actually had a fixed screen size because it shipped on the iMac, but there were a lot of cooks in the kitchen as to what the interface should look like. (Reid, CHM oral history, 1:35:18)

The designer was Priscilla Shih, hired from Fractal Design.

She and I came up with this scheme. Basically we knew we'd have to keep reprogramming the interface, so we designed a Photoshop file format. We used the layers in Photoshop to represent, for example, a button. You have a mouse-over state and a click state, they change colors a little bit, so those would be four layers in Photoshop. And we made up a little language in the Photoshop layer names, like this is the mouse-over state, this is the click state. We sort of recreated Interface Builder inside Photoshop files, if you will. And then we taught iMovie how to open and read in the Photoshop files to display the interface. (1:35:47)

Interface Builder was NeXT's tool for laying out interfaces visually; Reid had come from the company that made it. The point of the scheme was who could change what.

Because the artists were always changing the bits on us and we would have to keep redoing it. So we were able to give them the Photoshop file and say change it all you want, just don't mess up the layer names, because that's necessary. And we could read the entire, I think there were about 300 layers in the Photoshop file, and we could read the whole thing in and display it in half a second. So it was like a really good way to do user interfaces. (1:36:33)

A development convenience, until it wasn't.

We decided to ship it that way. So built into the first four versions of iMovie was a Photoshop file where we changed the file type so you never knew it was a Photoshop file, and would just read it in and that was your user interface. But if you knew that that was a Photoshop file you could literally open it up in Photoshop, change it and save it again, and iMovie would, you could skin iMovie if you knew that. (1:37:03)

It got better.

In fact you could do it while iMovie was running, because we needed that. So iMovie every two seconds, every one second, would check the file modification time on this Photoshop file, and if it had changed it would dump the UI and reload it. And so we could in real time play around with the user interface while the app was running, and move the button and change the title on it and stuff. It made it much easier for us to do our work, but we didn't really want people to be re-skinning it and screwing around with it, so we just didn't tell anybody. (1:37:28)

Reid says on tape it is the first time he has told the story.

Passing one Safari benchmark with a branch to an absolute address

Engineering · Objective-C garbage collection, Apple, around 2006

Objective-C made programmers manage memory by hand, and getting it wrong was the main source of crashes on the Mac. Garbage collection promised to handle it automatically, at a price: a collector has to watch memory reads and writes, and that watching costs time on every one of them. Blaine Garst, who had built NeXT's runtime and was now at Apple, was the person shipping it.

The gatekeeper was named, and so was the test.

I did a cardinal sin when I introduced garbage collection at Apple. In order to make it fast, I had to prove to Scott Forstall that it wouldn't slow down Safari. (Garst, CHM oral history, 2:55:19)

Scott Forstall ran Mac OS X applications. Safari was the benchmark because it was the application whose speed users noticed most, and a garbage collector's overhead shows up on exactly the kind of tight loops a browser runs. Garst needed the cost of the check to be invisible, and there was one way to make a check that cheap.

The garbage collector intercept point, I broke a cardinal rule. I had the compiler issue a hand-coded instruction to branch to a routine at an absolute address. So how many people get to deploy absolute-addressed entry points in application code? (2:55:38)

Normal code never jumps to a fixed address; the operating system decides where things live, and hard-coding a location is the sort of thing that gets a patch rejected. On the PowerPC there was an instruction that could do it, and it could only reach two places.

The instruction was branch and link absolute, BLA, like my name, Blaine, almost. And it could only branch to the top part of core or the bottom part of core. Bottom part of core, address zero, was taken. So we branched to the high-end address. There was a page or two of possible branch points for that instruction. So we took over the highest page of memory. (2:56:02)

Then the swap.

On the launch of any process we do a little quick check. Are we running garbage collected or not? And if we were, we'd swap out the entry points up there that would go off and do what we call read barriers and write barriers for garbage collection. And if you weren't running garbage collection, it would simply return. So it was a branch and link and return immediate, which was imperceptible in the Safari measurement. So we got to do garbage collection. (2:56:24)

Every memory access in every compiled program got a jump to the top page of memory. For a program not using the collector, the routine there did nothing and came straight back, fast enough that Safari's numbers did not move. For a program that was, the same address held the real barrier code, installed at launch.

The feature shipped in Mac OS X Leopard. Apple deprecated it a few years later in favour of a different approach.

A crash converted into a beep, and a linker fix nobody would make

Engineering · The original Macintosh, 1983 to 1984

The first Macintosh had 128 kilobytes of memory. Applications did not fit in it, so their code was split into segments that were loaded when needed and thrown out when not. Two stories from that constraint, told by the two people who lived closest to it: Bill Atkinson, who wrote QuickDraw and MacPaint, and Andy Hertzfeld, who wrote most of the Toolbox.

Hertzfeld sets the frame, and he does not flatter it.

We were really fighting against a very, very small amount of memory. That's because memory was one of the most expensive components and we wanted our computer to be affordable to ordinary people. So we worked really, really hard to get it to fit in a tighter spot than it really could. And so a lot of the fragility in Mac programs was caused by, we were really shoving more functionality in there than we could fit. (Hertzfeld, CHM MacPaint oral history, p.13)

Atkinson has the number for MacPaint.

I know that MacPaint in its worst case had 134 bytes free. And I could drive it to that state by loading the right big font. And all that and one of the challenges is how do you prove that you've found the worst case use? (Atkinson, p.13)

The answer to that was a tool that came out of something else entirely. Steve Capps had been asked to build a recorder so the team could make tutorials that played the computer back to itself.

So make a mechanism where he could record events and feed them back. So he had the brilliant insight well, you don't have to record the events to feed them back. (Hertzfeld, p.13)

You could make them up. (Atkinson, p.13)

You could just make them up. And so that became a measure of robustness of Macintosh applications is how many minutes or hours could they survive the monkey. And eventually after a lot of work they could last all night. (Hertzfeld, p.14)

The Monkey typed random keys, picked random menu items, and dragged random tools across the screen, forever. Anything that leaked memory on any path would eventually die under it. Atkinson: "MacPaint went two weeks once." (p.14)

That leaves the failure the Monkey could not prevent: fragmentation. Enough memory in total, but no single hole big enough for the next segment.

When code segments were loaded, you needed some code to do this job and that job. They would be loaded sort of at the first available place. But if you needed another code segment and this one would go out, it might leave a hole there that wasn't quite big enough for the next one that you needed but now you had sort of what we called memory fragmentation. That even though you had enough memory total, you couldn't load the pieces of code that you needed. (Atkinson, p.14)

Atkinson's fix did not fix it.

I developed a little technique for this which is setting a flag at the top of the event loop, saying that we failed and as I went to load code segments, if I failed to load one then I would beep and let it go back to the top of the event loop without doing anything. [...] The net result was the user would go to draw something and it would beep and they would try it again and it would work, and they'd shrug and they'd never know that they just avoided crashing the program. (p.14)

A segment fails to load. Instead of crashing, MacPaint beeps and returns to waiting for input. The user tries again, memory has shifted, it works.

The other story is about the machinery that did the loading, and it is in Hertzfeld's separate session, cut from his book as too technical. The segment loader needed a small change to the linker, which ran on the Lisa and belonged to the Lisa team.

I needed some support from the linker, which runs on the Lisa. I didn't have access to the linker source code. Just to make a simple change, just off multiplying something by eight instead of six when it was building something. So, there was no technical limitation. You know, there was nothing hard, but the guy responsible on the Lisa side for doing that hated the Mac, and wouldn't do it [...] I think I never tried to get Steve on it, just because of its technical nature, but I had to come up with a horrible hack that made everything work without them changing it. After the Mac shipped, the guy finally made the change for it, but it was precarious what I was doing. I knew it was, but I got away with it. (Hertzfeld, CHM oral history part 1, p.33)

Moving the menus to the top of the screen, and the second invention that paid for it

Design · Lisa and Macintosh, 1980 to 1983

The mouse was new enough in 1980 that basic questions had no settled answers, including how far the pointer should travel for a given movement of the hand. Bill Atkinson, who wrote QuickDraw and much of the Lisa's interface before moving to the Mac, worked on those questions at home on one of a handful of Lisa prototypes and brought the results in as Polaroids.

I had a Lisa prototype in my home and there were only a few of them, and I would do user interface development at home and then ride my motorcycle in to Apple and show them, show Polaroids to the Lisa team. I took Polaroids and I would bring them in. And so I actually have sort of chronicled in those Polaroids a step-by-step how we bumbled through the user interface for the Lisa which became pretty much what we had on the Mac. (Atkinson, CHM MacPaint oral history, p.4)

One of the steps was where menus live. Every graphical system since has put them at the top of the window, the top of the screen, or both, and the choice is not cosmetic.

At one point I had, I moved the menus from being, I think at that point they were at the top of the window, I moved them to the top of the whole screen so I could always have the full width. If the window was stubby you didn't have to lose any menu titles. And also I'd always have the full height. If the window was down near the bottom, what do you do? Are you going to bounce it upwards or how do you get the menu to show right? (p.4)

Then the effect nobody designed for.

Because the titles were short and the items were long it kind of multiplied your screen real estate by like three times but you sort of didn't realize it. They just appeared. A given menu item was always at the same place on the screen. It was sort of a kinesthetic thing. You could go and move to and start the action before you even sort of, you know, totally aware of what you were doing. (p.4)

Andy Hertzfeld adds the mechanical reason it worked: "Because they were at the very top, you didn't have to aim. It would just, you would just go all the way up." (Hertzfeld, p.4)

The Lisa's application writers did not see it that way.

When I first moved them from the top of the windows and the Lisa application writers screamed. They said "That's going to be so much farther to reach because then we can't, you know, we can't just reach up to the top of the window. We'll have to go all the way to the top of the whole screen." (Atkinson, p.4)

The objection was correct. The fix was a second invention.

I think this is about the time we incorporated variable speed mouse scaling. [...] That if you moved quickly you got a different gear ratio. So millimeters on the table to millimeters on the screen, you'd move more millimeters on the screen if you were moving quickly and that way you'd make sort of a quick upward move and it would move all the way to the top easily and pin there because there wasn't anywhere else to go. [...] It was actually easier than the top of the window because you could overshoot the top of the window. (p.4-5)

And the second invention turned out to be necessary on its own.

That also solved this problem of how do you point between two lowercase "I"s? You need sort of the mouse to go into compound low where a lot of millimeters on the table makes a few millimeters on the screen so you can very carefully point between them. Now I remember the Lisa team saying "Oh this is not going to work. The mouse is going to fall in your lap." You know if you move up quickly and down slowly, eventually it will fall in your lap. And it sort of does but nobody notices. Every so often they pick it up and reposition it and it worked. (p.5)

Hertzfeld, at the demo machine during the interview: turn the scaling off in the control panel and try to use the Mac without it. "I don't think you'll last more than a minute." (p.5)

Stopping work on the Finder to build the database underneath it, against orders

Engineering · The original Macintosh, 1982 to 1984

Bruce Horn came to Apple at 22 from Xerox PARC, where he had spent his teens in the Smalltalk group. He wrote the Finder, the program that shows you your disks and files, and he wrote the Resource Manager, which almost nobody outside Mac programming has heard of and which is why the Finder, and everything else, could exist in 128K.

The Finder started as a demo of something that did not exist.

I had an idea of the files and folders idea. [...] I thought I would do that instead, and the idea of you could open a folder and look at it and close it and so on, I did a demo of that actually from not even looking at the file system, but just looking at a text file that kind of described what a file system might look like. So it read in the text file and you could kind of navigate this. And that was my demo of what I think it should look like. And then I ended up writing the real Finder after that. (Horn, CHM oral history, p.13)

A fake file system in a text file, navigable, shown as "what I think it should look like." Then, on the way to building the real one, he stopped.

Because I kind of think of foundational things, I was thinking, "Well, how am I going to store the icons? How am I going to store the strings?" and so on. And one of the things that I cared about was I wanted to make these programs be able to be done in different languages and not have to rewrite the code, right? So I wanted to separate out all the language stuff from all the code itself. And I started thinking about, well, if I had a little object-oriented database [...] So I ended up going off of my track. And I kind of stopped working on the Finder and working on the fundamental part that would support the Finder and everything else with this little object-oriented database, which was called the Resource Manager. (p.13)

Resources are the parts of a program that are not logic: every string, icon, menu and dialog layout. Keep them separate from the code and you can translate a program to Norwegian, or redraw its icons, without recompiling it. In 1982 that separation was not standard practice, and Horn's version of it had to fit in what was left of the ROM.

I only had three k-bytes of assembly code to work with. That was what was left in the ROM to build an object-oriented database in assembly code. So it worked out, but it was a struggle to make it fit. (p.16)

His manager, Bob Belleville, did not agree that the visible deliverable should wait for its substrate.

Bob and I disagreed on the importance of the Resource Manager. And I just wouldn't yield. I basically said, "No, we have to have this." Right? Because I was counting on it for the Finder. Bob thought, "What's this computer? It's just going to be another computer down the road and another one after that. What's so special about this Mac?" And I thought it was, here was my chance to really do it right, the way I thought it should be done. And so I basically just told him, "I have to do this." And I had backup from Andy and also from Steve to finish the Resource Manager. (p.15)

He got the backing, and he did not make the date.

I do remember at one point promising something about being done in two or three months, and I put stickies up day by day all around my cube, three months' worth. And I'd pull one down every day. And Andy recalls that after the last one was pulled down, I kept working on the Resource Manager anyway. But by then, it was obvious that it was the right thing to do. (p.15)

Why it was obvious: Hertzfeld rewrote much of the Toolbox to use it, so windows and menus and dialogs all lived in resources, and with memory that tight the ability to load and unload objects on demand was not a nicety.

I don't think it would have been possible to do, have the internationalize-able system that we had. I think that we probably wouldn't have been able to do the complex programs that we'd had, because the Resource Manager would allow you to kind of swap in and out objects as you needed them. And we were so constrained with memory that we needed that capability. (p.16)

Then he finished the Finder, with Steve Capps pushing it over the line at the end.

The Finder was 46K bytes. Forty-six K bytes, right and the Lisa Filer that they had done, they kind of redid the Filer after they saw the little pre-demo that we did of the Finder, that I did of the Finder. That weighed in at, like, 360 K bytes. So ours was only 46 K. (p.18)

Why you could name a Mac file anything you liked

Design · The original Macintosh, 1983

Every other system in 1983 identified a file by its name. The letters after the dot told the machine what kind of file it was, and if you renamed it wrong the machine lost track. The Mac did not work that way, and the reason is a design decision Bruce Horn made while building the Finder, which he describes as coming straight out of how he had learned to think at Xerox PARC.

I had this idea of a Type and a Creator. [...] The concept of a Type and a Creator was unique and new to the Mac. Where the Creator would basically be what an application would export as, "This is who I am." And the Type was, the file would say, "This is the kind of thing I am." And you would bind those together. (Horn, CHM oral history, p.13)

Two four-letter codes, stored with the file but not in its name. The Type says what the file is. The Creator says which application made it and should open it. The Finder kept the binding between them in a database of its own, so double-clicking a document knew where to go without asking the filename.

So the Type and Creator system was the thing that allowed you to, for example, name a file anything you like, and all of the type and creator information would be hidden away in the file system that Larry [Kenyon] did. And that was just a bonus. We wanted to name anything you wanted to name and type it with spaces, and we didn't want to have filename suffixes or any of that stuff. We just wanted it to be very natural. And so Type and Creator came out of that. (p.13)

The interviewer asks whether the plan from the start was no extensions, just names. Horn's answer is that the question is backwards.

No, no. These things are objects, right? They have types. They're of a particular kind, right? We want the file to be an object. And so with the file as an object, where do you keep its type, and if it's going to be manipulated by an application, how does it know? And all that had to be part of the system that was built and supported by the Resource Manager, the Type and Creator, and Desktop Database. (p.14)

I grew up thinking object-oriented because I came out of LRG. And I wanted to do as much as I could that would make the system be as dynamic as it was in the Smalltalk world. Of course, you can't do that if you've got a tiny, tiny computer, your operating system is all in assembly code in ROM. How do you make things dynamic? And the best I could do was this object-oriented database. (p.14)

LRG was PARC's Learning Research Group, Alan Kay's Smalltalk team. Horn had been inside it as a teenager.

The design lost, eventually. Mac OS X adopted extensions, because the rest of the world had them and files had to travel.

Choosing the processor so they could keep the graphics library

Engineering · The Macintosh, 1981

The original Macintosh design, under Jef Raskin, used a Motorola 6809, an 8-bit processor cheap enough for a machine meant to sell for under a thousand dollars. The Lisa, Apple's other project, used the 68000, a 16-bit processor with far more power and a price to match. Bill Atkinson was writing QuickDraw, the graphics library, for the Lisa and its 68000. The Mac team's plan was to rewrite it for the 6809. Andy Hertzfeld describes how that plan died.

Bud not only made the decision to use QuickDraw, he made the decision to use the 68000 so we could run QuickDraw. Bud was Bill's best friend. They went to college together. They both went up to Seattle together for grad school. [...] Bud was writing, essentially taking the QuickDraw design, but rewriting it for the 6809 chip that the Macintosh was using at the time, but every evening, seeing what Bill was doing on the 68000 and said, "Boy, I sure wish we could use that. You know, we'd save a year or something like that, if we could do that." (Hertzfeld, CHM oral history part 1, p.19)

Bud Tribble was the Mac's software manager. He was doing the port himself, which is how he knew what it would cost, and he could see the finished version running every evening on a machine he was not allowed to use. The problem was the price.

So he told Burrell, "Hey, isn't there some way we can get the 68000?" Well, we can't for the price point we were aiming at. We could only have one row of RAM. Then Burrell was brilliant, brilliant guy, was inspired, he saw a way to do it, to run the 68000 with only an eight-bit data bus, which was impossible, because it had 16 data lines. But Burrell devised an incredibly clever scheme that he called the bus transformation circuit, which not only allowed him to use the only eight chips, but it ended up being twice as fast as the Lisa design. So, it was both 1/3 the price and twice as fast, this 68000 prototype that Burrell came up with. (p.19)

Burrell Smith designed the Mac's hardware. A 16-bit processor wants 16 memory chips, one row per bit, and the budget allowed eight. His circuit let the processor talk to half as many chips and came out faster than the Lisa, which had all sixteen.

Once we had the 68000, it was obvious we were going to use Bill's QuickDraw as much as we could. I had to make changes to it to adapt it to the Mac's memory management model, which was different than the Lisa's. So there were, I would say, just we used 90 percent of what Bill did, just straight out. (p.19)

The prototype also changed who was interested in the project.

Jef hated it. And as soon as Burrell came up with the 68000 prototype, Steve got very interested in it. Jef was not interested, but Steve was interested in it. (p.19)

The keyboard that lost to layouts that worked better, and why

Design · The iPhone, 2006

Before 2007, typing on a handheld meant a physical keyboard, a stylus, or pressing a number key several times to reach a letter. Nobody had shipped a good keyboard on glass. Bas Ording had designed much of the Mac OS X interface and was now doing the same for the iPhone, and the keyboard was the part he was least sure could be made to work.

We knew it was going to be tricky to get it to work. We had already done some keyboard stuff on the bigger touchscreen. And so definitely I go like, "Oh, now it has to be on an even smaller screen, like this is going to be, this is not going to be easy." So it took us a long time, and like other engineers worked on it. And like come up with ideas, like all kinds of different things just to see, "How can we make something that actually where you can type on it?" (Ording, CHM oral history, p.44)

The interviewer notes there was a contest: keyboard design was opened to the engineers, and everybody built one. Ording confirms it, and says what the best entries had in common.

There was some interesting ideas, or stuff that worked really quite well, where you can't do it wrong really, but then it would be awkward, you'd have to do too much taps or too many swipes or it would just look too weird that you don't recognize it as a normal keyboard. (p.44)

Layouts existed that were more accurate on a small touchscreen than QWERTY. They made errors nearly impossible. They lost.

I think in the end, Steve decided that it needed to be like a QWERTY keyboard, of course, if that can work well, because as you see it, "Oh, yeah! Keyboard!" No big deal, everyone knows what that is. And you know how to type it. You don't have to learn anything. It's like you'd have to be a little more careful, I guess, but then there's like a bunch of smart people that figured out like clever algorithms, how if you type a word, that it makes the best out of it and even if you're a little off, it still corrects it and all that, so it ends up working pretty well. (p.44-45)

Ording, on the state of mind before the correction software worked: "At first we thought, 'I'm not sure how this is going to work at all!'" (p.45)

The three decisions Hertzfeld got wrong, in his own list

Engineering · The original Macintosh, 1984

Most accounts of the original Mac's software are about what its authors got right. Andy Hertzfeld, asked in 2025 about the decisions that hurt the platform later, gives a list, and each item on it was made for the same reason.

The most obvious one, I would say, was running in supervisor mode on the 68000. That was indicative of, get our drive to simplicity. We basically would remove complexity in the model and just make it simpler for the developer, really, if you just didn't need to manage that. That was wrong, because we were going to evolve into a system that needed more protection. We got rid of some of that basic protection, just because we wanted to keep it simple, but that was too simple, I think. (Hertzfeld, CHM oral history part 1, p.23)

Supervisor mode means every program ran with full authority over the machine. There was no wall between an application and the system, so a bug in one could take down everything.

Another bad blunder I made, was using low memory like the Apple II did. I just saw the way the Apple II did the operating system and thought that was good enough and cool. You actually, by putting the global variables in the memory, you save memory space, because the first 32k of memory, the 68000 could address just in a two-byte address instead of a four-byte address. So, I thought that was a good idea, until you considered, well, we wanted to run two programs at once, they both would fight over the memory variable, so that was not a good decision. (p.23)

Another one like that was we added some features, I did myself, to the memory manager. [...] So we put the flag bits to control these new features. Was this object purgeable from memory, etc. We put them in the block header because, well, the 68000 had a 32-bit address, but we were only using 24 bits of that in the model. So, we thought we could use those higher order bits as pointers, simplify things and make things a little faster. That was a disaster a few years later, because we couldn't expand memory past what turned out to be an insufficient limit, and so, we could fix that by changing the system to put those bytes in a different place, but it caused chaos and messed up the early developers, some of them. (p.23)

The processor could address more memory than the machine had, so the spare bits in every pointer were used to store flags. When Macs got enough memory to need those bits, every program that had touched them broke. Hertzfeld had told developers not to. "But people did anyway."

The interviewer offers the summary: decisions that seemed reasonable because the machine had to be simple and cheap, and bit the platform later. Hertzfeld goes further than agreeing.

We didn't know what we were designing for, as history has proven. We thought we were trying to make an exquisite product that's as fast and as inexpensive as possible, but that that would be replaced by an entirely different one within three or four years, just as the Mac was replacing the Apple II, we thought something would come along and replace the Mac. We had no idea that it would last, you know, some of these basic architecture things would last for a decade. So, what we were really doing was designing the first of a long-running platform, but we didn't understand that when we were building it. [...] We hyper-optimized to the machine, where it was in 1984, and it would have been better to make it a little less efficient, a little more memory consumptive, to give it an easier transition into the future. (p.23)

Building overlapping windows because he thought he had seen Xerox do it

Engineering · QuickDraw, 1979 to 1983

Overlapping windows are harder than they look. If one window partly covers another, the machine has to know exactly which pixels of the back window are visible, which means computing an arbitrary shape and clipping every drawing operation to it, fast enough to keep up with typing. The alternative is to redraw everything from back to front each time, which is simple and slow. Bill Atkinson wrote QuickDraw, the graphics library under the Lisa and the Mac, and he solved the hard version for a reason Andy Hertzfeld tells on his behalf.

First, why it mattered. Atkinson on the state of the art:

When you have windows you've got sort of one window and another one obscures parts of it and another one. So when you draw into one of the behind windows you have to clip it to some arbitrary area made by the shapes of the other ones that are overlapping it. And there were ways to draw clipped information of the SIGGRAPH core stuff, you sort of pre-divided the line segments and things like that. But they were only about two orders of magnitude too slow to do a real word processor with. (Atkinson, CHM MacPaint oral history, p.2-3)

A hundred times too slow. Then the famous 1979 visit to Xerox PARC, where Apple's engineers were shown the Alto.

Should tell the funny story about how during the Xerox PARC demo, Apple got one demo from the Xerox stuff, and Bill thought that he saw them drawing into behind windows, windows that weren't the topmost window. And so he knew it could be done and he worked really hard to make it. It turns out that was erroneous. They weren't doing it. He solved, he was motivated to solve the problem because he thought he saw an example but it really was a fundamental problem and they hadn't solved it but Bill did. (Hertzfeld, p.3)

Atkinson's version, including the conversation with PARC afterwards:

Sometimes a little ignorance can be very motivating, and later I think one of the people from PARC said "How did you do that?" "What? I thought you were doing it?" "Oh, no, no. We were drawing the back, we'd have to redraw all the other windows." "Oh, I missed that point." (Atkinson, p.3)

The thing he built is the data structure at the centre of QuickDraw.

So I was able to come up with a very efficient way that really didn't cost much more than drawing into a rectangle but could allow for very arbitrary shaped objects that are clipping. (Atkinson, p.3)

The very center of that in the heart of QuickDraw's capability was the status structure called a "Region" and since you guys are going to have access through the MacPaint source code through this website, you can look at the region code and see how lovingly it was designed and brilliantly it solves that problem. (Hertzfeld, p.3)

Hertzfeld's summary: "It helps to be young to make breakthroughs because you don't know what's impossible." (p.3)

Inventing fuzzing three days before the NeXT launch

Engineering · The NeXT Computer launch, October 1988

The word did not exist yet. Testing software meant writing down the cases you had thought of and running them, which by definition leaves out the bug you have not imagined. Feeding a program random garbage until it breaks was not a recognised technique. Avie Tevanian arrived at it under a deadline.

Tevanian had written the Mach kernel at Carnegie Mellon and joined NeXT to work on the core of its operating system. The NeXT Computer was to be launched in San Francisco in October 1988, and it shipped with a magneto-optical drive instead of a hard disk. Three days out, the driver for that drive did not work reliably.

I have a vivid memory of the lead up, of a part of the lead up to that, which is two or three days before the launch, we were working on the optical drive. And so, I was part of the OS team. [...] I wasn't responsible for the optical driver. I was responsible for more of the core of the OS. And it was working pretty well. But the optical driver was unreliable. And I remember the guy working on it, outstanding engineer. His name is John Seamons. He was burning the midnight oil just trying to get this thing working in about three days before the intro. And this has got to work. I'm like okay, I've got to roll up my sleeves and help John. (Tevanian, CHM oral history part 1, p.47)

The first move was not to look for the bug. It was to build a way of finding out whose bug it was.

I said, "Okay, we've got to do something. We've got to start building reliability tests for the file system, so we can figure out, is it a driver problem, is it the file system, whatever." And so, I built a very simple UNIX level test to read and write files in a slightly random way and ran it. And it failed immediately. And I built in logging and stuff, so we could then figure out what was going on. We went, and we tested. And we looked at logs. We figured out the bug. Great, we fixed it. Good, we're all set to launch. (p.47)

They did not stop there.

Before we do that, let's take the testing to the next level. And so, he and I kept iterating on these test programs to the point where we finally had a test program, which was basically issuing random file system instructions to the OS, completely random, no program would ever do this, just to get to it to fail. And we would see these failure modes. Oh yeah, that's a bug. And oh yeah, that's a bug because even though it's random, it should still work, right? And so, we finally got it all to work. And we, I think the two of us worked almost continuously for forty-eight hours to make that work. (p.47)

Asked whether it was stressful:

I don't remember it being stressful. I just remember being, "Wow, this is so cool." [...] These are the hardest kinds of bugs to find. [...] I remember I learned a lesson from that, which is all of us engineers think we can write code that works. But what you've really got to do to test your code is throw garbage at it and then see what happens. When you throw things at it that is what you expect, yes, it works. But a lot of times, developers throw things at it that you didn't expect. (p.48)

The launch went ahead. The demos, in Tevanian's account, all worked.

Exposé came from noticing what the Dock's animation implied, and its trigger from watching where people parked a button

Design · Mac OS X, around 2003

Bas Ording had been hired to design the Mac OS X interface off a demo, and his tool was Macromedia Director, an animation program in which he wrote small working prototypes. Exposé, the feature that spreads all your windows out so you can see them at once, started as the kind of problem everyone has and nobody had a solution for.

One day I'm just staring at my screen, and I have a bunch of windows and they're overlapping each other. And I'm thinking like, or I was sort of imagining like, "I wish I could just sort of like go behind that window, so I can just see them all." [...] "Well, let's do this, start really simple. What if I just have two windows, and one is above the other one? What would you do to see them both?" I'm like, "Well, I can just like separate them. That would be easy to do. And then if the windows are too big, if they won't fit in the screen, I can just like scale them down." [...] "Oh! well, if it's three windows, then well, it gets a little more complicated, because how are they going to move?" (Ording, CHM oral history, p.11-12)

He worked out an algorithm: shuffle the windows apart iteratively until none overlap, then scale the whole arrangement to fit, and animate straight to the result. A prototype in Director, and no way to run it against real windows. The way in was noticing something about a feature that already existed.

I worked together with John Louch, from Engineering, and he's super good. And he worked on the Dock to get that all to animate, like high frame rates and get it really working really well. And I knew that he could, because of the Genie effect, that was owned sort of by that Dock, the process that runs the Dock and that has to basically grab a window and morph it, right? Or and manipulate it. So I knew that like John Louch knew how to access the windows on the screen. So I'm kind of like, "Hey! If you can grab these windows, you can grab any window, right?" "Yeah, yeah!" "So, okay, well, let's run this algorithm on these rectangles, basically, and see what happens." So we started to do that, and of course, like his code went like ten times faster than my code. (p.12)

The Genie effect is the animation that sucks a window into the Dock when you minimise it. To do that, the Dock had to be able to take hold of any window on screen and distort it. If it could do that to one window on the way to the Dock, it could do it to all of them for any purpose.

So because of John we could get all this stuff to work in the real system, too. And we could just play with it and like really use it for real. And then we could demo to Steve Jobs and a bunch of other people, and they got all excited, and then, of course, you got all the other ideas. "Oh, yeah, it needs to be just the app windows!" Or, "It needs to be all the windows!" (p.12)

Then how you trigger it, which was not designed at all.

For a while we had, this is silly, this like round blue button that we had sitting on the screen that you could drag around anywhere. If you click it, it would just do Exposé. If you click it again, it goes back. But this button was kind of in the way. It was kind of useful, but also in the way. So you end up putting it all the way in the corner, and then we were like, "Well, if we put it in the corner, we may as well just use the hot corners like you use for your screensaver as well, right?" So that's what ended up being one of those features. You just put your mouse in the corner, and all your windows go, "Psht!" (p.12-13)

They shipped a draggable button into the live system, everyone who used it moved it out of the way to a corner, so they deleted the button and kept the corner. Asked whether any of this went through the user testing lab: "No, we weren't. No. [...] For those things, we didn't really do that." (p.13)

The iPod started with a drive nobody wanted, shown at the end of a meeting about something else

Product · The iPod, 2000

Jon Rubinstein ran Apple's hardware engineering, and by 2000 he had been trying for a year to find the parts for a music player. The idea was not the problem. In his telling the storage was.

We needed a small, cheap display. We needed a small, cheap battery. We needed a processor, right, and we needed storage and so I started scrounging around for all this stuff and trying to put the pieces together. And I was hanging out at Almaden at IBM Research because I was friends with Nick Donofrio and Nick said, "Come on by. Take a look at our technology. See if there's anything you want to use." So I took a look and I saw the Microdrive when it was under development and I reached out to the Microdrive Group and said, "Hey, I need five gigabytes and this price." And they laughed at me. Right? "Never going to happen." (Rubinstein, CHM oral history part 1, p.87)

In 2000, storage meant a spinning disk, and disk sizes were set by what laptops needed. A drive too small for a laptop had no market, which is exactly why one was sitting unsold in Japan.

I kept looking and eventually I was in Japan and found the Toshiba drive and they weren't really sure what to do with it because it wasn't, it didn't have enough capacity to really go in a PC. Right? And I was there and they were taking us through their roadmap for all of our other products, and at the end of the meeting, they go, "We have this other thing. Would you like to take a look at it?" I'm like, "Yeah, sure. I'll take a look at it." So we looked at it, and like it's obvious, right? This is how to make an iPod, right? (p.87)

A supplier roadmap meeting for other products, a drive mentioned as an afterthought.

Jeff Williams is there with me, and I'm like, "Jeff, we need to get all of these." So Jeff goes, "Yeah, we think we have an idea. We can use this. How many of them can you make, and can we get exclusive?" And he started the whole exclusive conversation, so I, Steve was in Tokyo, so I said, "Steve," and I've told this story lots of times but, "I need a $10 million check to go do development on this." And Steve goes, "No problem, I'll write you the check." So then I talk to Fred to make sure the check won't bounce because Steve, Fred was the guy who actually wrote the check, not Steve, and Fred goes, "Go ahead. The check won't bounce." (p.87-88)

Jeff Williams, then in operations and now Apple's chief operating officer, opened the exclusivity negotiation on the spot. Fred Anderson was the CFO.

The rest of the parts were already there, for a reason Apple had nothing to do with.

The cellphone industry had really started taking off with, remember, cellphones used to have single-line displays, and it had just been recent at that point in time that the bigger displays, black and white, they were not color, they were black and white, displays had started becoming available, same thing with battery technology. Right? The lithium ion battery technology had just started getting incorporated in cellphones, so there was this convergent of great technologies. Right? And the form-factor was self-evident. It was going to be about the size of a pack of cards because if you took the display and the battery and the circuit board and the hard drive and the connectors you needed on either side, that defined the form factor. (p.88)

One more detail from the same passage, on which the people involved do not agree.

Phil Schiller is the one who came up with the scroll wheel. He had a B&O phone, an old B&O phone that had the scroll wheel, right, and they didn't have acceleration in those days. But, so we grabbed that idea. We patented it right away, and we actually ended up licensing that patent back to B&O. (p.88)

Tony Fadell, who was hired to run the iPod's engineering, gives the same account of the wheel in his own interview and adds a hedge about the years since. On the larger question of whose idea the iPod was, the two men disagree flatly.

Rubber-banding was a debugging aid before it was a design

Design · The iPhone, 2005

Before the iPhone, a list on a screen was moved with a scrollbar, and when it reached the end it stopped. Dragging the content itself with a finger did not exist as a convention, so there was no convention either for what should happen when the content ran out. Bas Ording was building the first scrolling list for the phone, in December 2004 or soon after, and the answer came from a problem with his own prototype.

I remember working on this demo where you scroll through the list and I had lots of times where because I was constantly changing stuff in my code, like, I thought I was running the code, but then it wouldn't scroll and I'm like, "Oh, I guess I'm not running the code yet. Like, wait." But then it was running. But I'm like, "Oh, wait, I'm scrolling in the wrong direction because it's at the top of the list and it's not going to go any further," so I would, it just wouldn't move because it was at the top. (Ording, CHM oral history, p.25)

Two failures looked identical. A prototype that had not started, and a list that had reached the top, both did nothing when you dragged. He could not tell which he was looking at.

And then I started to think, "Oh, maybe I need to add some white space or something that you can see that it's at the top, right." But then still, if you try to move it, and nothing will happen, it feels sort of weird as if the program is just stuck or something. And so that's when I started to think about what can we do to make it feel still alive somehow, that it has some kind of give to it. (p.25)

The fix for his debugging problem was to make the list respond even when it could not move.

One of them was, like, adding this sort of this, yeah, thing where it moves but only at like half the rate of your, that your finger is moving, just to get the feeling of, like, of some kind of resistance and, and then if you let go, it would just spring back to the top of the list. And then from there, like, if you were to scroll at higher speeds and it reaches the end, and it bounces back a little bit and all that. So all the behaviors that go with it, basically to make it feel right. So it took a little while, but it was fun to work on. (p.25)

Every touchscreen since has done it.

I remember, yeah, showing that to Steve and he got all super excited about it, so that was cool. (p.25)

"Interpersonal computing" on machines that could only talk to each other

Marketing · NeXT, 1987 to 1989

Dan'l Lewin ran Apple's higher-education sales, left with Steve Jobs in 1985 as a NeXT co-founder, and ran NeXT's sales and marketing until he quit in 1989. His account of how NeXT positioned itself is the account of the person whose job it was to sell the positioning, and who did not believe it.

The framework came out of a board meeting.

The IBM PC was blue, right? So we had blue, red, and green. And so blue was IBM, and that was all about Lotus 1-2-3 and spreadsheets. And that's what made the PC. And red was Apple, and that was all about desktop publishing. And that's what that was going to be. We were going to leapfrog them all, and we were green. And we were going to be interpersonal computing. So, spreadsheets made the PC, DTP the Mac and Interpersonal computing would be NeXT. (Lewin, CHM oral history, p.25)

Each previous platform had been made by one application category. NeXT's would be communication between people: voice attachments on email, a demo one engineer had built in a week. The problem was the installed base the positioning assumed.

The whole idea was this radical new thing to Steve that people actually were using networks. And we were going to position ourselves as interpersonal computing, when a NeXT machine could only talk to a NeXT machine. I look at it and go, no, this is like a good idea in the shower, but a bad idea in the market. Only our machines can talk to our machines, really? (p.25)

This was 1987 or 1988. Lewin dissented in the room and the positioning went ahead. Then the price. NeXT had been founded to sell a $3,000 computer to students. The 1988 launch priced the cube at $6,500 to universities, and configured it came to around $10,000. Lewin's account of the retreat where the cheaper machine was born says what the gap did to the company.

When Steve came in, and he was sort of morose, and he was down, and I was sitting there, and I sort of said, "you know, we built this company to sell a $3,000 computer. We have a $10,000 computer. Why don't we think about designing a $3,000 computer that people could use and that our market could afford to buy?" Damn, that's a good idea. Let's do that. And so Rich, and George, and Steve, and the whiteboards show up, and all of a sudden it's like, OK, how will we skinny this thing down? (p.30)

Why the price had got there is a design story told as a cost.

The cube itself was supposed to be burdened, fully finished, ready to have the board stuffed in. It was supposed to cost $50. The paint job cost $50. It was a magnesium thing pressed by... you know, it's like... (p.30)

Asked whether he tried to reason with Jobs about it: "Oh, of course! Of course, it was insanity." (p.30)

The go-to-market plan was an attack on Sun's distribution.

Sun had, they were about $400 million, right? They had just gone public. [...] For the most part, they were selling their hardware to an OEM who was bundling software on top and physically delivering the hardware and the software together, whether it was Frame, or Interleaf, or these CAD machines [...] So what we were doing is cherry picking all the software on top, getting those guys to migrate, have Businessland push the distribution out. All they would have to sell is their software. So their cost of sales would change. They wouldn't have to carry inventory, and we could build to order with our factory. So I was going disassemble Sun's entire channel play. (p.30)

Take the software vendors who resold Sun hardware, move them onto NeXT, let a retailer handle the boxes, and Sun's channel collapses. It needed a $3,000 machine. It had a $10,000 one.

We had $120 million or so in the bank, and it was glaringly obvious to me that he was going to burn through all of it in about 12 to 15 months, and there would be no stopping him. The board meeting, I quit the week after this board meeting where Perot blew up and quit. Everybody was done with him. I mean, he had 51 plus percent control of the company, and he was taking no input from anyone. (p.30)

Fifty thousand Macs that dealers would not take, sold to universities instead

Marketing · The Macintosh launch, 1984

The Macintosh shipped in January 1984 into a dealer channel, and Dan'l Lewin's job was the other channel: universities, through what became the Apple University Consortium. It is remembered as a programme to put computers in front of students. Lewin's own account is about inventory.

Because we did the introduction in January, and things sort of frittered along for a while, I was really looking at summer deliveries for fall, for the academic year. And we actually saved, we, our little program, there's a few people in the field that helped me, but it was really my program at corporate [that] saved the company's ass on inventory penalty costs, because the dealer network topped out at about four units a month per dealer. And the factory was running. (Lewin, CHM oral history, p.26)

Four units a month per dealer, against a factory that had been built for a much bigger number.

I think we had about 20,000 units in inventory at launch. And maybe the build rate was about that per month, about 20,000 a month. But it took a bunch of months to get to that point. But then the ramp was going up. And they kept building. And the dealers would only take replenishing their initial shipment, and so all that inventory that got built I took and put into the universities. And so we shipped 50,000 computers in the university community like that. (p.26)

The consortium's big orders were real and were coming: Drexel at 3,000, Dartmouth, and one that arrived as a place in line rather than a delivery.

There was like the purchase order from University of Texas where Charlie Warlick came to me and said I want to get my place in line. And he gave me a PO for 30,000 computers. When do you want those, Charlie? As soon as you can get them to me. You know, and they took a lot of them, but they didn't take 30,000 out of the chute. (p.26)

That period let us turn back the inventory for manufacturing RAM because the penalty clauses and the lead time, especially in those days, was immense. So we both made a lot of money. The margins were extraordinary through that program, but I also saved the company a lot in that period. (p.26)

Memory was ordered months ahead under contracts with penalties for cancelling. A factory building 20,000 machines a month that dealers absorbed at four each was about to trigger those penalties.

The customer who took the Mac apart and told Apple what its network should be

Marketing · Apple and Dartmouth, 1983 to 1984

Before the Mac launched, Dan'l Lewin and Mike Boich were taking prototypes to universities. At Dartmouth, an engineer opened one up.

He looks inside. And Mike and I are sitting there, and he goes, is that an RS-422 chip? And we went, yeah. And so he's, wait a minute. He left the room. A couple minutes later he came back in with the manual for the RS-422 chip. And he's like looking at the manual. And he's in the corner with someone else. And then they go whisper at Bill Arms. And then Bill looks at me and he goes, uh, we got to go for a walk. (Lewin, CHM oral history, p.24)

RS-422 is a serial communication standard, faster than the one in most personal computers of the time. Dartmouth's people had recognised the chip and worked out what it meant before anyone at Apple had told them. On the walk, with the provost, they made an offer.

We're going to tell IBM, who want to give us PC Juniors, and tell Digital, who want to give us DEC Rainbows, to take their gifts and go home. And we recognize you can't give us these computers, because you're just a small company. But can you give us a lab. (p.24)

Lewin had learned one habit from Floyd Kvamme, Apple's marketing head: ask the people who buy from you why they buy, because they will tell you the truth. So he asked.

They said it's very straightforward. And the provost is with us, and he said we're going to make sure that every student has one of these. If they don't have the money, we'll build it into financial aid. [...] Well it's an intelligent graphics terminal that we can put on our network. At which point I said, but there's no networking built into this machine. And they said that's OK. You have a high speed serial chip. We will write our own networking. And I went, and how will you, and they said, through the telephone infrastructure. (p.24)

Dartmouth was the home of BASIC and of time-sharing on mainframes. They did not want a personal computer; they wanted a terminal for the network they already had, and they had found the chip that let them make the Mac into one. Apple's own networking did not exist yet.

We learned that from Dartmouth. And we didn't have AppleTalk, because remember, when Steve got fired, it's partly because AppleTalk didn't work. (p.24)

Then the phone-jack story. Jobs is known to have lectured Bob Metcalfe, the inventor of Ethernet, that nobody would buy networking until the connector was as simple as a phone jack, pulling the phone cord out of the wall to make the point. Lewin heard the story from Bill Krause, 3Com's CEO, years later.

Steve did his Steve thing and got up and walked over, and went over, and "it's got to be like this." And he pulls the RJ-11 jack out of the wall on the phone. It's got be like this. And I just looked at Bill, and I just said, well let me tell you where he got that. Because it was from my trip to Dartmouth where they just said we'll use the phone line. (p.24)

Think Different, as it was actually sold to the room: a schedule and a media buy

Marketing · Apple, September 1997

Think Different is remembered as a piece of film. The tape of Jobs introducing it, eight to ten weeks after coming back, is a different document: a working presentation to Apple's own people that spends its first three minutes on products and inventory before it gets to marketing at all.

We're trying to get back to the basics of great products, great marketing and great distribution. And I think that Apple has pockets of greatness but in some ways has drifted away from doing the basics really well. So we started with the product line. We looked at the product roadmap going out for a few years and we said a lot of this doesn't make sense and it's way too much stuff and there's not enough focus and so we actually got rid of 70 percent of the stuff in the product roadmap. I mean I couldn't figure out the damn product line after a few weeks. (Jobs, Think Different introduction, 0:29)

Then distribution, with a number.

We've got anywhere from two to three months of inventory in our manufacturing supplier pipeline and about an equal amount in our distribution channel pipeline, so we're having to make guesses for five, six months in advance about what the customer wants and we're not smart enough to do that. (2:16)

Only then, marketing, and the argument for what the campaign would not do.

The way to do that is not to talk about speeds and feeds. It's not to talk about MIPS and megahertz. It's not to talk about why we're better than Windows. The dairy industry tried for 20 years to convince you that milk was good for you. It's a lie but they tried anyway and the sales were going like this. And then they tried "got milk" and the sales have gone like this. "Got milk" doesn't even talk about the product. [...] Nike sells a commodity. They sell shoes. And yet when you think of Nike you feel something different than a shoe company. (4:14)

He had fired the agency and the 23-agency review Apple was running, hired Chiat/Day, and given them eight weeks. The film is a minute long and the room saw it once. What the tape preserves that the film does not is the buy.

We are breaking this campaign this Sunday in a rather poetic way. The Wonderful World of Disney is restarting on ABC and the first thing they're showing this Sunday night, I believe it's at 7 o'clock, is the network premiere of Toy Story, and we're gonna have two 60-second spots. This commercial will run twice, once in the first hour and once in the second hour. We are then gonna break some newspaper ads in the Journal, the Times, the Mercury, the Examiner, USA Today, really stating the manifesto, the words. [...] This ad will run throughout most of October on television and we're breaking some phenomenal print within a few weeks, mostly on the back covers of magazines, some on the inside. We've got some incredible billboards and we're even painting some giant walls in about five or six major cities. (10:54)

Toy Story was a Pixar film; Jobs was Pixar's CEO. The premiere was the launch slot. Then the permissions.

In this day and age to use any of these people, whether alive or dead, you need major permission from them, either themselves if they're alive or their estate's representatives if they're dead. Almost all of these people have never appeared in an advertisement before and never would until we asked them. I mean I got permission from Yoko Ono a few days ago to use John. (12:01)

And the close.

Now, advertising is not everything and we've got some incredibly exciting product announcements coming up soon. [...] The question now is not can we turn around Apple, I think that's the booby prize. I think it's can we make Apple really great again. (15:03)

The digital hub was a name marketing put on something that had already grown

Marketing · Apple, 2000 to 2003

The digital hub is taught as a strategy. In January 2001 Jobs stood on stage and said the Mac would be the centre of a person's digital life, the place cameras, music players and camcorders plugged into, and the applications Apple built over the following years are read as the execution of that plan. Glenn Reid ran two of those applications, iMovie and iDVD, and his account from inside is different.

Asked whether the creation of the applications group had been planned:

The creation of the iApps group just happened organically, as you said, and then led to this sort of digital hub strategy that Steve talked about. But it was not planned from the beginning. It was just kind of, it happened semi-obviously by following the media path. (Reid, CHM oral history, 2:39:55)

The media path: photos, then movies, then music, each a thing people were starting to bring home in digital form, each needing something to do with it on a Mac.

Apple was also growing and hiring people and acquiring companies and, you know, there were getting to be a lot of people, and actual marketing people. And so how do we sell this mishmash of weird products? So the marketing people sort of put wrappers around it and called it the digital hub, and the iApps and all those names. They were labels for things that already had organically come into existence. (2:40:23)

They basically came built in. It was a pretty powerful reason to buy a Mac at that point. There was a lot of pretty useful stuff that came with it. (2:41:18)

Boston, August 1997: arguing a booing room out of the position his own company had taught it

Leadership · Macworld Boston, 6 August 1997

Gil Amelio had been removed as Apple's CEO four weeks earlier, no replacement had been named, and Steve Jobs, officially an adviser, was running the company in everything but title. Apple's public identity for fifteen years had been built on Microsoft as the enemy, and the audience at Macworld Boston was made of the people most invested in that. This is the keynote where he announced a Microsoft investment and told them the war was over.

He opened by quoting the press back to itself.

When I started to get involved, a lot of people gave me advice. And some of the advice that was the most popular was Apple has become irrelevant. There was a great one that was Apple can't execute anything. And another one was the Apple culture is anarchy, no one could manage it. You've read all these things in the press. And after four weeks, here's what I found. Quite the opposite of these things, actually. (Jobs, Macworld Boston 1997, 6:49)

Then the conversion of each charge.

Apple is executing wonderfully on many of the wrong things. The ability of the organization to execute is really high, though. [...] They're doing some of the wrong things because the plan has been wrong. And lastly, what I found is rather than anarchy, I found people that can't wait to fall into line behind a good strategy. There just hasn't been one. (7:29)

Then the number, on one slide.

Apple's sales in 1995 were 11.1 billion. In '96, they were 9.5 billion. And in this year, they'll be, you know, 7 billion plus or minus a little bit. That's the problem or the symptom, depending on how you look at it. (8:50)

He replaced the board, keeping two members and adding four. The first boo of the morning arrived on a name.

The first is Larry Ellison, CEO of Oracle. I hope that wasn't a boo I'm hearing. (11:39)

Then market share.

I can't get anyone to tell me the definitive market share number for Apple, but it's around 7 percent from all I can gather. And the question is, where is Apple relevant? [...] It's like 80 percent of the computers used in advertising and graphic arts, design, prepress, all Macintoshes. And 64 percent is the best number I could find. 64 percent of all internet websites are created using a Mac. (18:49)

Who is the largest education company in the world? [...] Apple is the single largest education supplier in the world. [...] Now, I've asked 100 people at Apple this and only two have thought of it. It's incredible the position Apple still has in education. 60 percent of all computers in education are Apples. 64 percent of computers teachers use are Apples. It's a two to two and a half billion dollar business for Apple every year. (20:52)

On the Mac OS.

Most people think that we're about to abandon the Mac OS. [...] Macintosh OS 8, which we just released, was code named Tempo, as you know, right? So, we just released Tempo. Most people think our next release next year, which is code named Allegro, will come out. But then our next release will be Requiem. And it's crazy. (24:13)

Then the deal, in four parts: a patent cross-licence covering everything filed for five years, Microsoft Office on the Mac for five years at the same release cadence as Windows, Internet Explorer as the default browser, and a $150 million investment in non-voting shares locked for three years. The browser was where the room turned.

Apple has decided to make Internet Explorer its default browser on the Macintosh. Since we believe in choice, since we believe in choice, we're going to be shipping other Internet browsers as well on the Macintosh, and the user can of course change their default should they choose to. (28:47)

He restarted the sentence twice over the booing. Bill Gates appeared on the screen behind him by satellite, which the audience booed as well. Then:

Where we are right now is we're shepherding some of the greatest assets in the computer industry. And if we want to move forward and see Apple healthy and prospering again, we have to let go of a few things here. We have to let go of this notion that for Apple to win, Microsoft has to lose. Okay? We have to embrace a notion that for Apple to win, Apple has to do a really good job. And if others are going to help us, that's great. Cuz we need all the help we can get. And if we screw up and we don't do a good job, it's not somebody else's fault. It's our fault. (33:18)

I think if we want Microsoft Office on the Mac, we better treat the company that puts it out with a little bit of gratitude. We'd like their software. So, the era of setting this up as a competition between Apple and Microsoft is over as far as I'm concerned. (34:08)

Then a reframe.

Another bolt of lightning is that Apple plus Microsoft equals 100 percent of the desktop computer market. And so whatever Apple and Microsoft agree to do, it's a standard. (35:18)

The fifteen-point plan delivered uninvited, and the whiteboard binary from years before

Leadership · Apple, 1996

Regis McKenna had shaped Apple's marketing from the Apple II onward and had long since left the company's payroll, but not its orbit. By 1996 everyone in Silicon Valley was having lunch with him about Apple.

A lot of people from Apple and from other places were having lunch with me, or talking to me, and saying, "Something ought to happen at Apple because it's dying." There was just no pizzazz in the company any more in terms of products. The products like their pen-based and voice-based systems, all these were failing. Newton and things like that had failed. [...] I even had people calling wanting to have lunch with me who said that they felt that they could come in and run Apple. (McKenna, CHM oral history part 5, p.7)

He did something about it that nobody had asked for.

I formed an alliance with my firm and Gemini, the consulting group. Some people from Gemini came to me and said, "Something ought to be done at Apple." And so we sat down together and came up with what could they do. We spent an afternoon on a whiteboard doing a session on it, just for our own mental health, I guess. And out of that I wrote a plan as to what they could do in 10 or 15 different steps. (p.8)

Point one was not a product.

One of the first was profitability. I strongly feel that the best positioning tool that any company in the technology business has is being profitable. If you're profitable then the world will look at you and say, "Okay, you're doing something well and right." If you're not profitable they want to know what's wrong, and the CEO gets put on the spot. The company also becomes very vulnerable when you're losing money. It's when your competitors start hiring your people because they figure you're vulnerable. It's when your competitors start going after your customers because they know you're probably not going to be a long-term presence in the market. (p.8)

Then the last point.

Number 15 I remember well. I said, "Bring Steve Jobs back into the company and give him a strategic position of helping to straighten out the company and set his vision." I don't think he thought about that, but that was number 15. (p.8)

He took it to Gil Amelio, Apple's CEO, without an appointment.

I gave him a presentation, probably about two hours in his office, point by point. It involved building alliances with other companies out there and learning from them, that sort of thing. [...] He didn't react much at all to the whole thing. (p.8)

McKenna's read on why:

I think he also relied heavily on his staff. And there was a real arrogance, I think, among his management. I learned later he had developed that at National, we know how to do this thing and nobody else does. And they sort of were, "Go away. Go away. Go away." (p.8)

Its fifteenth point happened a year later, by a different route. From the same session, years earlier: at an Apple staff meeting, McKenna had drawn a fork on a whiteboard.

That's where, in my notebooks, I show the split [in direction] that I had shown them. One was towards IBM. The other was to be a Sony. I said they had to make that kind of decision because it would mean how they staff, how they hire, how they market, everything. [...] I even drew a little triangle, and I put down, "What are the requirements to get into either one?" [...] Sure enough, when Steve came back, he was quoted in Time Magazine as saying, "We want to be the digital Sony." (p.6)

The ledger: promises with dates from the Mac OS X unveiling, and how they aged

Leadership · Macworld San Francisco, January 2000

Keynotes are promises with dates on them, and almost nobody checks. The January 2000 unveiling of Mac OS X and its Aqua interface is checkable, because the developers Jobs brought on stage said specific things about how long their work would take.

The schedule, stated as a rollout.

There's going to be a 12-month rollout of Mac OS X. You can't do these things overnight. It's gonna be a 12 month rollout so we are announcing it today, January 2000. Our developers have already had a few betas of the software. (Jobs, Macworld SF 2000, 1:36)

He then listed the stages: a developer release that month, a beta in the spring, on sale in the summer, preloaded on every Mac a year out. Fifty-five minutes later, closing the segment:

Mac OS X, going to be rolling out on a Macintosh near you the second half of this year. (56:27)

Mac OS X 10.0 went on sale on 24 March 2001. The public beta appeared in September 2000. The three-tier developer story, and its cost estimates, were also stated on stage.

There are three of them in Mac OS X: Classic, Carbon and Cocoa. And the reason for this is to provide a general migration for people from the left to the right over time. [...] Classic runs Mac OS 9 apps as is without modification. We have Carbon which allows the developers to tune up their Mac apps really fast and get all the features of X, and then we have our very advanced object oriented APIs, Cocoa, which lets you write apps in a tenth of the time. (4:30, 22:10)

Then the partners. Adobe first.

Adobe just had an incredible year. We surpassed the billion dollar mark. Steve, if it wasn't for you it wouldn't have happened. Almost half of our revenue, close to 500 million dollars of our revenue, was because of sales of our Macintosh applications. [...] We are absolutely committed to OS X. (Adobe executive on stage, 47:10)

Microsoft committed to two products on day one and left the third open.

Microsoft is fully behind Steve and his OS X product. We're going to be there on day one with Internet Explorer 5 and Outlook Express 5 for OS X. We've got a great new version of Office in the works to ship later this year. We'll go to OS X on that as soon as we can as well. (Kevin Browne, 49:00)

Office for Mac OS X shipped in November 2001. Then two developers, one after the other, with opposite estimates for the same task.

We just got a new system in about a month ago and we decided to try porting our most popular application, which is Flash. 200 million people have Flash Player today. And so we decided to port it, so put an engineer on it, one engineer for a week and a half, and it's running. (Rob Burgess, Macromedia, 51:30)

I wish I could say that ours was only going to take a week and a half. QuarkXPress however is a bit more of an effort. We've been getting a lot of help from Apple and we really appreciate that, but we're going to make it there. (Quark, 52:40)

Flash was a small, self-contained program. QuarkXPress was a decade of page-layout code that the entire publishing industry ran on. QuarkXPress did not ship natively on Mac OS X until 2003.

And one promise from an earlier keynote, with the counterparty standing next to him.

When we announced our partnership with Microsoft two and a half years ago most people couldn't understand it and didn't think anything good would come of it. And I got to tell you, we're working very closely with these guys. Kevin has an awesome team up there of Mac developers and they're doing some great applications on Macintosh. (Jobs, 49:33)

The Boston 1997 bet, closed on stage in 2000 with Microsoft's Mac chief beside him.

Shipping a Unix workstation whose only disk took a third of a second to seek

Product · The NeXT Computer, 1988 to 1989

The NeXT cube shipped with a magneto-optical drive and no hard disk. The drive held 256 megabytes on a removable cartridge, far more than any floppy, and looked like the future of storage. Rich Page ran NeXT's hardware.

The problem was not capacity or reliability. It was time, and what a Unix system does with a disk.

Imagine swapping onto something that has a 20 millisecond seek time versus swapping onto the optical that probably had a seek time of three, four hundred milliseconds. So I mean the optical was good but it was not the best thing to swap to. That was kind of a misuse. (Page, CHM oral history part 1, 2:31:45)

Unix moves memory to and from disk constantly as part of normal operation. Every one of those moves waited a third of a second instead of a fiftieth.

The original cube did not ship with a hard disk. [...] I don't know how long we shipped the optical only but it was like six, nine, twelve months. [...] In the minimum configuration we were shipping it without the hard disk to get the price down, but we came to the realization it's really not very usable without the hard disk. So then that's when we put in the small hard disk. It's sort of an accelerator for swapping. (2:32:10)

Page connects it to the Lisa.

There's a similar problem with the Lisa, right? [...] What happens is you pick a price where the margins are fairly low but livable and you start there and you make the assumption over time you can bring the cost of product down, you know, and maybe hold the price. But the problem is if you have to add anything to the product to make it more usable, that drives the price up and takes time to bring the price of the product down. So those two things work against you. (2:33:29)

Bricking the best iPhones in existence three weeks before the keynote

Engineering · The iPhone launch, December 2006

The phones Steve Jobs would hold on stage at Macworld in January 2007 were hand-carried from China in steel cases, by people who flew there, collected the suitcases, and flew straight back. Each one was graded. Andy Grignon, who ran the iPhone's radio engineering, describes the grading and what his team did to the best of them.

Steve Jobs and Jony Ive would put jeweler's loupes on and they would put gloves on and they would grade the quality of each phone, right, based on the kind of defects it had, because when they were still bringing up a line, plastics don't sit right. They're not meshed or they maybe have a scuff during some bad assembly process, so they're given a grade. So, double As are effectively production level devices. As are really good. Bs, Cs, and Ds fall, like Cs basically just go to QA because they're visually problematic. So, we had I think eight double As and then I forget how many units we had that were As. (Grignon, CHM oral history, p.24)

A new batch had come in around Christmas, and Grignon's team had a software fix to load onto them. They used a gang programmer, a rig that flashes eight or sixteen phones at once.

They go through the programming sequence and they reboot, and all of them as they're coming back up, they crash, crash, crash a bunch of times, and then finally they all go dead. (p.24)

What had happened was inside the radio chip, and it was a feature.

The chip that made the phone calls was the boss, and the security feature in this chip would detect if somebody was tampering with it by squirting their own code onto the chip that would maybe subvert some security policy of the network, maybe make free phone calls [...] After a certain number of iterations, which happened very quickly, if it detects that I've been tampered with, the security feature literally lights a piece of metal on fire inside of itself and it renders it useless. So, it is a fuse that is blown deep in the heart of this chip, and since that chip is dead and since it's the boss, the phone is dead. It's not like you could just lift that chip off, put a new one on. The whole phone's dead. (p.24)

An anti-tamper fuse, designed to stop people stealing phone service, physically burned by a bug in Apple's own update.

We burned up our double As, and some of our As, the best phones that we had. These were supposed to be the phones that Steve is onstage holding that were flawless, the best that we could produce at the time, and we lit them all on fire, well, that little tiny piece of metal. And I thought for sure I was gonna be fired after that. (p.24-25)

Why no test had caught it:

There was a bug in our code that was triggered only when there was an existing software on there already. So, it worked fine if you were just a QA person that had a previous version, you flashed a new build, then it was fine. But if you were taking a brand new piece of thing off of the floor and you put the software down, that's what triggered the security thing, which burned the wire. (p.25)

Every QA phone had been flashed before. The bug only fired on a virgin unit from the factory, which is the one state no QA process ever has a phone in.

You'll see in some of the pictures at Macworld, there's these glass domes and they're really beautiful phones, and they're spinning around and they're off. Well, those are the best ones we had, and since there was high resolution photography, people could get up close to these things, we wanted those still to have that perfect, you know, the best that we could produce. They were supposed to be running a demo loop of just stuff that the phone, like a video that would just play over and over and over again, and a lot of them are off. (p.25)

The photographs from that launch are famous. In some of them the phones in the display cases have black screens.

What it cost the people who built it

Management · The iPhone, 2006 to 2007

Andy Grignon ran the iPhone's radio engineering. This is his account of what the project was like to be inside.

The setting was a room he named.

The people that made the chip, the baseband, were based out of Europe, and we had all these awful bugs in their stack, in our stack and nothing was working right, and it got to the point where we're like, "Look, we just need everybody onsite," and I had them bring like 20 or so people from Denmark and Germany and wherever. We flew them out long term to Cupertino and I shoved all of them in this windowless, cold server room. And we got to name our rooms and I called it European Vacation. (Grignon, CHM oral history, p.32)

A morning bug meeting, every day. One morning after being shouted at by Tony Fadell, who had been shouted at by Jobs:

The project manager sat at the complete opposite end, and we had gotten kind of comfortable with each other [...] he was just kind of leaned back and he had his legs crossed and he was like, "Eh." [...] For whatever reason I just lost it and I just, like a crazy person. I'm slamming my hands on the table, banging, and I just feel myself losing it, right? You know, that's not me, but that was the cauldron, that pressure cooker that we'd been put into. [...] I take my laptop and I threw it against the cinderblock wall [...] In my head it was like an explosion. Like, it hit the wall and parts are flying everywhere, and it was, like, this big, climactic scene, and then I storm out of there. And I'm sure really what happened was it just kind of like, thunk, slid down to the ground, very anticlimactic. (p.32-33)

His own diagnosis: "A lot of us had adopted Steve's temperamental management style, just the explosive, unpredictable, emotional." (p.32)

Then the argument in Scott Forstall's hallway, a week before Christmas.

It was basically like, "Well, if you don't want me to see my kids this weekend at their holiday show then I guess I can handle that now." It was some kind of quip like that, right? And this other person had kids and they're like, it became this surreal debate about who spent less time with their kids. And it turned into this screaming match between the two of them, and this other person storms off so angry, goes into her office and slams the door, just hard, like boom. [...] She slammed the door so hard that the lock mechanism broke and she was unable to open the door handle. (p.33)

The locksmith was an hour away. Forstall, who ran iPhone software, had been working late.

Forstall shows up. He had an aluminum bat and we're all taking full-size whacks at the door handle, at the lock on this door, just like boom, trying to get this thing unstuck. And finally it was Forstall who just went agro on this thing and he beat it to the point where it actually popped off. (p.33)

He follows the stories with this.

I think when you look at a project like iPhone, you know, it takes a pretty significant personal toll, professional toll. I mean obviously it was a successful product, but there was a lot of sacrifices that a lot of people made. (p.33)

At the end of the interview, asked about the ten years since:

What I say is I got a divorce because work is my mistress, right? One thing Steve was really good at was identifying people who will do anything it takes to ship a product, and that works for and against you, right? I'm your person to deliver a new cutting edge product. I will do that at the expense of literally anything else, my own health, my marriage, anything else, relationships. And some of these products, and the iPhone is one of them, requires that, not just from me but lots of people, and that was kind of a pattern when you look at everybody in aggregate, all kind of the same. [...] One of the jokes that we used to say was, the greatest thing about Fridays is it's two more working days until Monday. (p.56-57)

And, in the same breath: "I'm glad it was a success. I wouldn't take it back. I wouldn't do anything really differently." (p.57)

Where these came from

Most of these came from the Computer History Museum's oral histories, three-hour sessions with a few hundred views each, and the people in them are still adding to the record. Two of the sessions here were recorded in late 2025. Everything quoted is verbatim from the transcripts, with a page or timestamp so you can go and check. Where two people remember the same event differently, both versions are here and I have not picked.

If you want the interviews themselves rather than my selection, the companion guide lists every one with its video and what it covers. Start with whichever person you already have a question for.

"You can't connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future."

Steve Jobs, Stanford commencement address, 12 June 2005

On Monday, Apple gets a new CEO. The company is fifty, and it is entering AI the same way it entered every other new market: via consumer devices.

Between 1985 and 1996, Steve Jobs's NeXT sold on the order of 50,000 machines, closed its factory, and became a software company with $273 million in accumulated losses.

Apple bought NeXT in 1996 for $429 million, making it one of the most consequential acquisitions in tech.

What Apple really wanted was the operating system and the engineers, Steve Jobs, not so much. Within a year, the acquired company was running the acquirer.

A few ago, I fell into an Apple history rabbit hole watching the Computer History Museum archive. I found the interviews and stories fascinating and packed with lessons about management, engineering, design, and product.

I've organized the main timeline here as a rough, high-level history of Apple's Second Act — the phase that paved the way for what the company is today.

Apple wanted an operating system, not a CEO

What Apple wanted from NeXT was the operating system it had failed to build for itself – but the CEO who came with it took over the company.

Mike Markkula was Apple's employee number three and sat on the board through the acquisition. Asked what Apple was after, he did not hesitate: "what we wanted was the OS. And Avie Tevanian."

Whether Jobs returned was less important to the board. Markkula:

"I don't think it was an issue. If he wanted to come back, fine, if he didn't, fine."

Tevanian became Apple's head of software. Jon Rubinstein, who had run NeXT's hardware, became head of hardware. Bertrand Serlet, who had built NeXT's AppKit, later ran all of Mac OS X. Scott Forstall, a NeXT engineer, ran the iPhone software team. Every major Apple operating system since 2001 descends from NeXTSTEP.

Joanna Hoffman was on the original Macintosh team and followed Jobs to NeXT. In her 2018 oral history, she described just how complete the replacement was:

"It was NeXT's software that created that Macintosh to the exclusion of the previous Macintosh. People don't remember the previous Macintosh."

Her shorter version: "today's Macintosh success is NeXT. You know? No question."

Timeline of Apple's second act, 1985 to 2008

1. Sept 1985: Jobs, demoted from the Macintosh division, resigns and takes five people with him.

2. 1986: NeXT licenses the Mach kernel from Carnegie Mellon instead of writing its own.

3. 1988: NeXT Computer launches at $6,500 with a magnesium case and no colour display.

4. 1988: NeXT adopts Objective-C from Stepstone; Naroff moves it into GCC.

5. 1989: Canon invests $100 million.

6. 1993: NeXT closes its factory and becomes a software company.

7. 1996: NeXT drafts an IPO it never files; Apple kills Copland and goes shopping.

8. Dec 1996: Apple picks NeXTSTEP over BeOS and buys NeXT.

9. Feb 1997: Tevanian and Rubinstein take over software and hardware; the deal closes.

10. 1997: 300 R&D projects cut to about 50; Advanced Technology Group shut.

11. Aug 1997: Microsoft invests $150 million; Jobs becomes interim CEO.

12. 1998: The iMac, begun as a network computer, ships in fourteen months; Apple starts selling its ARM stake.

13. 1999: The iBook ships with AirPort; Wi-Fi in a consumer product three years before Dell.

14. 2000: Mac OS X's Aqua interface unveiled; Jobs drops "interim".

15. 2001: Rubinstein finds the Toshiba 1.8-inch drive; the iPod ships in eleven months.

16. 2001: Safari team picks KHTML over Mozilla.

17. 2005: Two iPhone projects compete inside Apple.

18. Jan 2007: The iPhone is announced; Apple Computer, Inc. becomes Apple Inc.

19. July 2008: The App Store opens with 500 apps.

1985: Why Jobs left, in the words of the two men who were there

The familiar version has John Sculley firing Steve Jobs. Neither Sculley nor the board member who investigated the dispute remembers it that way.

Markkula had recruited Sculley as CEO in 1983. The Macintosh Office, launched at the January 1985 annual meeting, "was a complete dud." By March, the disagreement had become a fight over the company's priorities. Sculley:

"As we moved out into March, the Macintosh sales were not doing well and Steve and I started to have major disagreements on what we should do about it. Steve wanted to lower the price of the Macintosh. And yet he still wanted to run substantial advertising behind the product. And he wanted to de-emphasize the Apple II."

Jobs did not think Sculley would take it to the board. Sculley did:

"I said if you try to change that on your own, then I have no choice but to go to the board, and we need to bring this issue up with the board. And he didn't think I would do that. And I did."

The board sent Markkula to interview the executives and decide who was right. Sculley:

"Mike Markkula did that, took him about 10 days to conduct this project. He came back, reported to the board, and his conclusion was that I was right."

The board removed Jobs as leader of the Macintosh division. Sculley came from corporate America, where executives get moved around. Only later did he understand what the move meant to the founder who had created that division:

"The board asked him to step down from the role of leader of the Macintosh division. To be quite honest, I didn't appreciate coming out of corporate America, because remember people get moved around all the time in corporate America, I didn't appreciate what it meant to a founder, the creator of the Macintosh, to be asked to step down from the very division that he created."

The distinction matters. Sculley:

"So Steve was never actually 'fired' from Apple, but he was demoted from the role of leading the Macintosh division and then he went off on sabbatical and then he eventually resigned from the company and took a number of key executives and started NeXT Computing."

Markkula gave the same account separately:

"John came to the conclusion that Steve wasn't working for him. Steve was doing what Steve wanted to do and it was disruptive. And I think he made the only decision he could make, which is, he can't manage the company if Steve's going to run around and undo things and change it and cause him all this grief. So he reorganized and gave Steve a job that didn't have anybody reporting to him, and he had an office away from the main campus. He didn't fire Steve, but he made it so Steve had nothing to do so Steve decided to go off on his own."

Markkula's criticism was about how Jobs left:

"I don't criticize Steve for wanting to do that or doing it even, but I just wish he'd done that himself, got something started and then thought about hiring people, whether they were from Apple or anywhere else and done it on the up and up. What he did was, he got half a dozen people to agree to go with him, and I felt that was unethical and not the right way to do it."

Rich Page was one of those people. He supplies the date. Jobs resigned from the board on 12 September 1985. Page:

"then the five of us turned our resignations in on the Friday the 13th, and then you get sued. Steve gets sued the next day, and it takes about six months to get rid of that."

The five were Page, George Crow, Susan Barnes, Bud Tribble and Dan'l Lewin. All had worked on the Macintosh.

1985 to 1996: What happened to Apple while he was gone

Sculley ran Apple until June 1993. His defence is numerical: the company was profitable and growing when he left.

"When I left Apple, we had about 2-billion dollars of cash, we had maybe 200 million dollars of debt."

He puts the damaging choices after his departure:

"After I was pushed out of Apple, Apple decided to license the technology and stop making one or two cool products a year, and made many, many different products and the company started to spiral into huge losses."

Regis McKenna, who marketed the Apple II and knew Jobs for 35 years, watched from outside. His assessment of Michael Spindler, the CEO after Sculley:

"He could stand at a board and draw out a future strategy, and it would overwhelm you. He fell short in how you implement it."

Gil Amelio took over in February 1996 and found the result. The R&D count still sounds like a typo. It was not.

"There were 300 R&D projects, 300. I could make a case that maybe we needed 20 of those, and that might even been too big a number, but 300 was ridiculous."

He got the count to about 50. The factories were no comfort:

"We were getting back, some months, as much as 10 percent of the computers we shipped."

Copland, the in-house replacement for the Mac OS, had been in development for years at "a couple of $100 million a year" with no end in sight. Amelio killed it. Apple now needed an operating system it did not have.

1986: Why NeXT licensed Mach instead of writing its own kernel

While Apple was losing its way, NeXT was making the decisions that would eventually replace its software.

Avie Tevanian was a graduate student at Carnegie Mellon working under Rick Rashid on a research kernel called Mach. NeXT's engineers saw it presented at a Berkeley Unix conference and decided they wanted it.

Tevanian and Rashid flew to San Francisco for the conference and learned that Jobs wanted to meet them. Tevanian:

"We go over to the NeXT offices on Deer Creek Road and we're sitting down in the lobby waiting for Steve, and Steve comes down all excited. 'Hi guys, how you doing?' Shakes his hand. 'Oh, I've met you before,' that's what he says to me. Like, no, you haven't met me before."

Tevanian also interviewed at Microsoft and Sun. The choice came down to Microsoft and NeXT:

"Microsoft was very organized. They wanted to hire me. And NeXT wanted to hire me. So I had to decide between those two."

Had he gone to Redmond, he thinks a different operating system history would have followed:

"If I had gone to Microsoft, NT probably would not have existed, or would have been something different. Because they basically hired Cutler instead when I didn't go there."

That is the decision that connects an academic microkernel to every iPhone. Mach became the kernel of NeXTSTEP. NeXTSTEP became Mac OS X. Mac OS X became iOS.

1988: The factory, and the cube

https://www.youtube.com/watch?v=dSj6kvv7_Sg

NeXT built an automated factory in Fremont before it had customers. Jobs commissioned the film above because a slide could not capture the automation. A board goes from bare to populated in about twenty minutes, with robots placing components and no hands on the line. It is mesmerizing. It is also a factory waiting for buyers.

Rich Page took over manufacturing in 1991. The cube's magnesium case was beautiful and unforgiving:

"The NeXT cube was a magnesium structure but painted, as it turns out. I think that was a mistake, because it's hard to take something that's cast, whether it's cast a little bit over, cast magnesium, it's hard to take somebody has cast, sand it, do it right, texture, paint it, get it the right color, and have that whole process be repeatable."

The second-generation NeXTstation used a flat case the team called the pizza box.

https://www.youtube.com/watch?v=92NNyd3m79I

The launch took place at Davies Symphony Hall on 12 October 1988. The footage was thought lost until the production team of the 2015 Steve Jobs film recovered it from two VHS tapes.

On stage, Jobs listed what universities had requested:

"give us 8 megabytes of RAM, give us at least 100 megabytes of local storage."

The industry called that specification the "3M machine": a megabyte of memory, a megapixel display, and a million instructions a second. The NeXT machine cost $6,500 and shipped with a monochrome display. The factory film plays at 29:58; the cube powers on at 42:15.

Universities buy in academic years, through committees, at fixed budgets. That is not a market that absorbs a factory's worth of volume. NeXT sold tens of thousands of machines against a plant built for hundreds of thousands.

1988: Why NeXT chose Objective-C

NeXT did not invent its programming language. It licensed Objective-C from Stepstone, where Brad Cox had grafted Smalltalk-style objects onto C.

Steve Naroff was Stepstone's engineer on the compiler. He fixed the language for NeXT's needs and joined NeXT in 1988 after Stepstone failed to resource the work. There he moved Objective-C into the GNU C compiler and added categories, which let developers extend classes they did not own. Blaine Garst and Bertrand Serlet added protocols. Garst added reference counting to memory management.

The peer-reviewed history is a paper Naroff wrote with Cox and CHM curator Hansen Hsu, "The origins of Objective-C at PPI/Stepstone and its evolution at NeXT", open access.

C++ won the industry. Objective-C had one serious customer. Then that customer had Apple.

1989: Canon's $100 million

Ross Perot invested in 1987. In 1989, Canon put in $100 million for a minority stake, the figure repeated in every account of NeXT.

Susan Barnes, NeXT's CFO, raised the Canon money. She has never given an interview about it. Of the five people who left Apple with Jobs, she is the one whose account does not exist, and hers is the financial one.

1993: The decision to exit hardware

On 25 May 1993, Jobs opened a NeXT keynote with a summary of the retreat from hardware:

"We set ourselves a 100 day transition to move from a hardware systems company to a software company. We've succeeded. Been transitioning to a software company in this 100 days. Our factory's been closed, the inventory's been sold, we've seen the last of black hardware. We've sold our service business to Bell Atlantic. Our hardware designs are being sold to Canon."

The product was now NeXTSTEP for Intel. Page had left in 1992, and by then every co-founder except Jobs had gone.

What NeXT bought that day was portability: an operating system that ran on someone else's chips. Twelve years later, Apple used it to move the Mac to Intel.

1996: WebObjects, and the IPO NeXT never filed

Most Apple histories describe NeXT in 1996 as a failed company that Apple rescued. The company's own paperwork says something different.

In November 1996 NeXT drafted an S-1, the filing a company makes before going public. It was never filed, so there is no record at the SEC. The draft survives because Tevanian donated it to the Computer History Museum, and Hsu wrote it up.

The first number is grim: $273 million in accumulated losses.

The next numbers complicate the obituary. WebObjects, NeXT's web application server, plus OPENSTEP had made $10.1 million in the first nine months of 1996, or 45 percent of software revenue. NeXT was preparing to go public as a web tools company.

Apple did not intercept a corpse. It intercepted an IPO.

December 1996: What Apple bought, and what it chose it over

Amelio went to Sun first. Scott McNealy wanted to buy Apple; the plan was to put the Mac interface on top of Solaris. Sun's board stopped it. Amelio:

"At the end of the day, Scott McNealy's board turned us down. At that time, John Doerr was on their board, and I think he was, perhaps, most vocal in saying, 'No, we shouldn't make this connection.'"

Then came Be. Jean-Louis Gassée, who had run Apple's products in the 1980s, had built a new operating system. Amelio:

"Jean-Louis Gassée approached us about the Be operating system, and I'm kind of glad he did because it gave us another contender in the stack."

The Apple Fellows had already reached the same conclusion. Larry Tesler, who came to Apple from Xerox PARC and founded its Advanced Technology Group, says the group debated porting the Mac OS to Intel around 1995, rejected it, and recommended buying an outside OS instead.

NeXT had the best operating system and the most troublesome association. Amelio:

"the NeXT operating system was the best choice out there. I would say virtually everybody felt like we shouldn't touch it because of Steve Jobs. Every one of my board members and most of my staff told me forget about it."

He kept talking to Jobs because the remaining options were not close:

"because if I couldn't get Sun, then the third, whatever was third on the list was so far down the list that it really wasn't a viable strategy."

Markkula confirms that Amelio made the call:

"Gil is the guy that came to the conclusion that we needed to buy an OS. The other one we looked at was Be. And Jean-Louis had done a really good job with that, but it wasn't finished."

The decision would end Amelio's own tenure. He knew the risk:

"I agonized more over the decision as to whether to bring Steve Jobs back more so than I think any other decision I made in my life."

Ten years after Jobs left, Amelio's reasoning was that "it was still his company."

From NeXT's side, the deal happened fast. Calls went in to Amelio, technical diligence meetings followed, "and in basically under a month we had a signed deal to be acquired."

Then Apple pre-announced a major loss. Tevanian:

"Right after signing the deal to be acquired, so now we're looking at late December, couple days before Christmas, '96, as that quarter is coming to a close for Apple, they had to pre-announce a major loss, and that kind of took us by surprise. We certainly didn't see that coming, just because we weren't watching the company at all. We knew the company was struggling, but we didn't know that it was that bad off."

The price was $429 million, announced 20 December 1996. What Apple got for it: the Mach kernel, the NeXTSTEP operating system and its OPENSTEP frameworks, the Objective-C toolchain, WebObjects, and the engineers who had built all of it. Apple's FY2000 10-K books the deal with a $375 million charge for in-process research and development.

Both part 1 and 2 of Avie Tevanian’s oral history are very interesting, but the second part is when he covers the NeXT acquisition forward:

February 1997: The NeXT people take over

At the WWDC 1997 opening keynote in May, Amelio is still CEO, Tevanian and Rubinstein already run software and hardware, and a young Scott Forstall does a demo. The recording is on the Internet Archive and has about a thousand views.

Jon Rubinstein had run NeXT's hardware. He started at Apple on 3 February 1997, the day the deal closed. Apple had lost $740 million in the preceding quarter. Rubinstein's estimate: "Yeah, we had one quarter left when I got there."

He puts the company's value at about $2 billion. "It's a trillion today, it was 2 billion."

Rubinstein's arrival was not advertised around the office:

"They kept me, in particular, hidden for three days because I was going to replace like five division managers, right, because we were going to collapse all these divisions."

Larry Ellison later said Jobs had planned the takeover all along. Rubinstein's response to the version told at Jobs' memorial:

"that bullshit that Larry spun, Ellison spun at his eulogy, at his funeral and stuff was nonsense."

Rubinstein leaves room for the possibility that Jobs kept a secret from him, but not much:

"Larry has been quoted as saying, 'No, no, no. Steve had this strategy of how he was going to do this, and he was going to get back to take over the company.' That is certainly not what I saw. Now, it may be that Steve kept it secret from me. Anything is possible, but we were pretty close at the time. And I know Laurene didn't want him to go back, right, because it meant he would disappear."

1997: Cutting products, and killing the research group

On 13 May 1997 Jobs sat on a stool at WWDC and took developer questions for an hour and eleven minutes. He held no title at Apple. The first question was about OpenDoc, a component technology Apple had been pushing for years, and he said he was for putting a bullet in its head. YouTube copies of this one get removed periodically; the Internet Archive copy is the durable one.

The cuts went beyond products. Larry Tesler had founded Apple's Advanced Technology Group. In 1997, he was asked to close it.

Tesler:

"In 1997, beginning of '97, when Steve Jobs was half back, Gil Amelio was still running the place but kind of doing what Steve had recommended he do, Avie Tevanian, who was running software at that time, came to me and said, 'We're shutting down ATG. People say, oh no, we're going to lose a lot of great ideas. Since you started it, would you go over there and find out what the great ideas are that are still there and which ones we should save and which ones we should kill?' So I was sent back there to kill the organization I had founded, but I'm glad he did it. It was the right thing to do."

The group had ideas. Relevance was the problem:

"They were all really cool, but they were irrelevant. There was one that was Microsoft Surface, so it was 1997 to 2012, so 15 years. It was 15 years ahead of its time."

July 1997: Amelio out

Fred Anderson was Apple's CFO. Tevanian recounts the moment Anderson called Ellen Hancock after firing Amelio:

"right after calling Gil he called Ellen Hancock, who was also being fired, to tell her she was being fired and she first said, he first said to Ellen, I think, something to this order, 'Gil is no longer CEO,' and she thought he was calling her to make her CEO."

McKenna kept a notebook. An entry from early January 1997, after Jobs met Amelio, records that Jobs:

"said they don't appear to want help. They didn't offer him a job. They didn't ask him to be a consultant. They didn't offer any position to him at all."

McKenna's explanation for why 1997 worked when 1985 had not:

"I think he was just so far better equipped to do that in 1997 or '98 than he was in 1985. He had gone through NeXT and putting that system together with the importance of software, and then Pixar, the professionalism of the staff and the management and how they did things."

Then, a few months later, the phone call. "His first words were, 'What do you think about my new board?'"

August 1997: The Microsoft $150 million

At Macworld Boston on 6 August 1997, Jobs announced the new board and then the Microsoft deal: cross-licensing, a commitment from Microsoft to release Office for the Mac, and $150 million in non-voting stock.

Bill Gates appeared by satellite on the screen behind Jobs. The room booed.

Six weeks later Jobs spoke to Apple employees about the new ad campaign.

https://www.youtube.com/watch?v=NiwJ26kc2YE

Jobs explained what Apple needed people to remember:

"To me, marketing is about values. This is a very complicated world, it's a very noisy world, and we're not going to get a chance to get people to remember much about us. No company is. And so we have to be really clear on what we want them to know about us."

The campaign would not be about specifications:

"The way to do that is not to talk about speeds and feeds. It's not to talk about MIPS and megahertz."

The spot would break that Sunday on ABC, around the network premiere of Toy Story.

1998: The iMac started as a network computer

The product that put Apple back in front of new customers began as something else. In 1997 Larry Ellison, Jobs's closest friend and about to join Apple's board, was evangelising the network computer: a cheap machine with no hard drive that ran off a server. Apple started building one. Jon Rubinstein, running hardware, had shipped a computer with no hard drive once before, at NeXT.

"I keep going, 'The NeXT machine with no hard drive didn't work out well. This isn't going to work well because it doesn't have the,' we talked about balance before. 'The networks aren't fast enough yet to have the kind of balance you need.' But Steve kept pushing forward with it, and Fred kept going, 'Wait a minute, we need an entry roadmap.' So finally, we had sort of a knock-down-drag-out one day and we decide that we're going to switch the network computer to becoming the iMac and we grow the enclosure a little bit and we stick a hard drive in it."

Fred is Fred Anderson, the CFO, who Rubinstein says "deserves a lot of credit because he's the one who kept going, 'We need to have an entry-level product.'" The motherboard came from the existing G3 desktop. The schedule was the other decision.

"Normal product development at Apple took three years, and I'm like, 'We got to get to one-year cycles.' [...] And everyone goes, 'That's impossible.' Right? And I said, 'No, no, no. We're going to redo all of our processes at the same time.'"

He used the iMac to install the development process Apple still runs: gates, milestones, manufacturing engineers in from the start, legal agreements signed before shipment. The last of those came from a display supplier whose panels were failing in the field at forty to fifty percent when he discovered the contract had never been finished. "I used the iMac as kind of the pipe cleaner, right, to clean out the pipe and to institute all-new processes around how we're going to do stuff at Apple." It took fourteen to sixteen months.

Two more decisions came with it. Rubinstein chose USB, then shipping on a million PCs that nobody used it on, over Apple's own ADB and over FireWire, which was too expensive for a low-end machine. And the floppy drive went.

"There were not peripherals out yet, right, but they were going to come, eventually. I mean, a million PCs had shipped with USB. They just, no one used it."

The peripherals had to be conjured. Apple was shutting down a billion-dollar printer business at the same time, and Rubinstein called Epson and HP: "Dudes, I got a billion-dollar printer business. Anyone want it?" Both built USB printers to match the iMac; Epson's came in the same colours.

The launch nearly did not survive the CD drive. During rehearsals Jobs saw the tray slide out and called Rubinstein screaming.

"He goes, 'But what's this tray that comes out?' I'm like, 'That's how they work, right? I mean, that's what all computers have.' He goes, 'Well, I want it to be like my car where you just, you have slot-load.' And I'm like, 'Well, they don't make those.' Right? And he goes, 'Well, this destroys the product.'"

It shipped with the tray. The next generation had a slot. Rubinstein's verdict on what the product did: consumer market share "which had dropped below 2 percent, was heading back up to 8-percent kind of range." And on the one part he still regrets: "That round mouse was terrible. Yeah, that was a bad product, right, and that was industrial design sort of trumping human factors."

1998: How the failed Newton kept Apple alive

Apple had co-founded ARM in 1990 to build the processor for the Newton, Sculley's project. Sculley says Apple held "47 percent" and describes the sale as one of two things that kept the company alive:

"I give Gil Amelio credit for two things, one was he sold the 47 percent that Apple owned of ARM for about 800 million dollars, or so I'm told, and that 800-million dollars was crucial to keeping Apple alive, and he brought Steve Jobs back. If those two things hadn't of happened, there never would have been the rest of the story."

Apple's own filings put it differently. The FY1999 10-K says Apple owned 42.3 percent of ARM in September 1997, and that the selling began in October 1998, more than a year after Amelio left. That month Apple sold 2.9 million shares and its stake fell to 19.7 percent. During fiscal 1999 it sold about 32.6 million more for net proceeds of about $245 million, booking a gain of about $230 million. In the first quarter of fiscal 2000 it sold another 5.1 million shares for about $136 million.

By September 1999 Apple had $3.226 billion in cash and short-term investments, up $926 million in a year. Some of that was the iMac. A meaningful piece of it was ARM.

Sculley's timing and total do not match the filings, and I have not tried to reconcile them.

The accounts agree on the consequential part: Apple sold most of its stake in the company behind the chip architecture it now uses in every Mac, iPhone and iPad.

1999: AirPort, the Wi-Fi Apple did not invent

Rubinstein ranks it above the iPod: "the one I'm almost more proud of is Wi-Fi, Wi-Fi base stations, right? And we created that, right? We didn't invent it, but we created it."

The problem was schools. The iMac had taken Apple back into education, the iBook was being designed as its portable sibling, and classrooms needed networking that did not involve opening walls.

"Schools are impossible [to] rewire because of asbestos. A lot of the schools were old, had a lot of asbestos, so you can't go digging in the walls. So we spent a lot of time looking at powerline, at wireless, at phone line, all the different kinds of networking. [...] Intel was doing HomeRF, which was sort of the more popular standard at the time. I mean, no one shipped it yet, but it had a lot of momentum going behind it. And the other one was Wi-Fi, which was mostly used for industrial control at the time."

Apple picked Wi-Fi against Intel's standard. The technology came from a group outside Amsterdam that had passed from NCR to AT&T to Lucent, selling cards for a couple of thousand dollars and base stations for ten thousand to factories.

"Tim Cook did a great job negotiating a deal with them. I mean, he really, we spent a lot of time together over there working on this, and he really convinced them to drop their drawers and give us pricing that was unbelievable so that we could do basically $100 plug-in cards and couple-hundred-dollar base stations."

The iBook team was told to add antennas and connectors to a product already in development, at a cost they resisted. The RF engineers came from the Advanced Technology Group Apple had just shut down; one intern in the Wi-Fi group went on to found Ubiquiti. Then the launch, at the Javits Center in New York in July 1999.

"We took an iBook, we put an accelerometer on it, right? We put Phil Schiller on top of it, and we threw him off of it to land on the airbag and then wirelessly we broadcast the data from the accelerometer on the screen. [...] And then, every fourth or fifth seat in the audience, we put an iBook below it. Right? And so then Steve goes, 'Okay. Everyone reach below.' And we had Apple employees in the audience, right, and everyone pulled out their iBooks and started surfing the internet wirelessly in the audience."

The venue's lawyer tried to stop the jump that morning; Apple's general counsel wrote a liability release on the back of her business card. Rubinstein: "that's the beginning of Wi-Fi, right, of commercial and consumerized, commercial, consumer Wi-Fi. Yeah, three years ahead of Dell." Apple shipped "$20 million worth of antennas and connectors" into its products before the standard had any traction.

Then Intel tried to change the rules of the unlicensed radio band in HomeRF's favour, and Rubinstein went to Washington.

"I spent a bunch of time in D.C. at the FCC and at the House Telecommunications Commission talking to people about why they shouldn't change the standard and why Wi-Fi was the right way to go and why you shouldn't let Intel change things."

What Apple did not do is the part he still argues about. With Wi-Fi winning, he asked Jobs for three or four engineers to port the base station's configuration software to Windows and sell AirPort into the PC market. "Steve goes, 'Over my dead body.'" Jobs's reason was that AirPort was part of what made a Mac worth buying. Rubinstein thought it was a multibillion-dollar networking business; he lost, and says the loss is why he and Phil Schiller fought so hard, three years later, to take the iPod to Windows.

On the name: "We named it AirPort Base Station but that wasn't because we thought it was going to actually be in airports. Right? I mean, that was never in our thinking. Starbucks, McDonalds, I mean, we never thought about it that way. Right? And anyone who says we did is fibbing, right, because we didn't think that big. Right? This was, how do we sell more Macs?"

2000: Carbon, Aqua, and the compromise that made Mac OS X shippable

Adobe and Microsoft would not rewrite their applications for NeXT's frameworks. Apple built Carbon, a compatibility layer that let existing Mac software run on the new operating system with modest changes.

"Carbon and Cocoa were both peers, if you will. There was, at least in the early days, there was no plan to sunset Carbon any time soon, and yet we wanted to make sure that people transitioned so you didn't really see new features coming out on Carbon whereas you did on Cocoa."

The cost was a decade of two parallel APIs. The benefit was that Mac OS X shipped with software on it.

Bas Ording designed the Aqua interface and, later, the iPhone's. Jobs recruited him with a phone call at home in 1998, when Ording also had an offer from MetaCreations.

Rubber-band scrolling, the little bounce when a list reaches its end, came from a bug. Ording was building the phone's scrolling contacts list:

"I was constantly changing stuff in my code, like I thought I was running the code but then it wouldn't scroll. I'm like, oh, I guess I'm not running the code yet. Like, wait, but then it was running. Oh wait, I'm scrolling the wrong direction, because it's at the top of the list and it's not gonna go any further."

A list that refused to move looked broken. Ording:

"feels sort of weird, as if the program is just stuck or something. And so that's when I started to think about what can we do to make it feel still alive somehow."

Serlet led the engineering. He had joined NeXT in 1989 following Jean-Marie Hullot from the French research institute INRIA. He ran Mac OS X until 2011, when he recruited Craig Federighi to replace him and left.

2001: The Toshiba drive, and the iPod

Rubinstein describes the iPod as a procurement story.

He needed a small hard drive. At IBM Research, he saw the Microdrive in development and asked for five gigabytes at his price. The answer: "They laughed at me. Right? 'Never going to happen.'"

Rubinstein kept looking. In Japan, he found a Toshiba drive whose makers did not know what to do with it:

"So I kept looking and eventually I was in Japan and found the Toshiba drive and they weren't really sure what to do with it because it didn't have enough capacity to really go in a PC."

The meeting had been about other products. The useful part arrived at the end:

"They were taking us through their roadmap for all of our other products, and at the end of the meeting, they go, 'We have this other thing. Would you like to take a look at it?' I'm like, 'Yeah, sure. I'll take a look at it.' So we looked at it, and like it's obvious, right? This is how to make an iPod, right?"

Jeff Williams, now Apple's chief operating officer, was in the room.

Then came the money:

"Steve was in Tokyo, so I said, 'Steve, I need a $10 million check to go do development on this.' And Steve goes, 'No problem, I'll write you the check.' So then I talk to Fred to make sure the check won't bounce because Steve, Fred was the guy who actually wrote the check, not Steve, and Fred goes, 'Go ahead. The check won't bounce.'"

Phil Schiller supplied the scroll wheel idea:

"Phil Schiller is the one who came up with the scroll wheel. He had a B&O phone, an old B&O phone that had the scroll wheel, right, and they didn't have acceleration in those days. But so we grabbed that idea. We patented it right away, and we actually ended up licensing that patent back to B&O. I don't think they really knew that we'd lifted it from them."

Tony Fadell, who ran the iPod division, tells the origin differently. Rubinstein:

"Tony has re-spun the whole story that he came up with the idea of the iPod and that Steve called him and, I mean, that's all nonsense."

Fadell gives his account in his own book and interviews. The two heads of Apple's hardware disagree on the record about who thought of the iPod.

2001: KHTML over Mozilla

Richard Williamson had joined NeXT out of Swarthmore. Ken Kocienda joined Apple in 2001. Together they built Safari.

In June 2001, Forstall asked Don Melton to start a browser team. Mozilla, the open-source descendant of Netscape, was the obvious base. Williamson chose KHTML, the rendering engine from the KDE desktop project, instead.

The size difference mattered on a team this small:

"Mozilla was the leading candidate at that point, the other leading candidate, but it was a million and a half lines of code, and KHTML 150,000 lines of code. And there you go, it's three guys."

Kocienda says the decision turned on:

"the utterly amazingly convincing demo that Richard made to get this code working on the Mac."

KHTML became WebKit. WebKit became the engine in every iPhone browser.

2003: Safari launches

Don Melton ran the Safari project from its first day, 25 June 2001, and has not sat for a CHM oral history, but he wrote the launch up on his own blog ten years later. Two posts carry this section: Safari is released to the world and Memories of Steve. Melton now publishes as Lisa Melton.

The secrecy problem in 2002 was hiring. Andy Hertzfeld had worked out what Melton was doing before his first day and kept quiet. One hire was too visible to hide:

“However, when I hired Dave Hyatt in July 2002, then guesses started flying fast.”

Hyatt had built Chimera at Netscape and co-created the project that became Firefox, both on Mozilla’s Gecko engine, so the industry assumed Apple was building a Gecko browser. Melton had chosen KHTML a year before Hyatt arrived.

Jobs opened the Safari segment of the 7 January 2003 keynote with “So, buckle up,” then stated the goal:

“Then he defined one of our product goals as, ‘Speed. Speed.’ So, I tensed up. Not that I didn’t agree, of course. I just knew what was coming soon: Demo time.”

Melton had sat through at least four rehearsals. At one of them Safari hung on stage because the whole network connection had failed; Kocienda found the cause, and IT set up a redundant link.

“And for the entire six minutes and 32 seconds that Steve used Safari on stage, I don’t remember taking a single breath. I was thinking about that network failure during rehearsal and screaming inside my head, ‘Stay online, stay online!’ We only had one chance to make a first impression.”

Then the slide naming the engine:

“Then Steve moved a new slide onto the screen. With only one word, ‘KHTML’—six-foot-high white letters on a blue background. If you listen to that video I posted, notice that no one applauds here. Why? I’m guessing confusion and complete lack of recognition.”

“What you also can’t hear on the video is someone about 15 to 20 rows behind where we were sitting—obviously expecting the word ‘Gecko’ up there—shout at what seemed like the top of his lungs: ‘WHAT THE FUCK!?’”

In Ken Kocienda’s book, Creative Selection, he describes Jobs setting page-load speed as the standard the browser would be judged by, Melton proposing a test program to measure it, and the Page Load Test Kocienda built to run against every change, with one rule: nothing that made the browser slower went in.

In the summer of 2002 Jobs wanted the status bar gone:

“Steve didn’t like the status bar and didn’t see the need for it. ‘Who looks at URLs when you hover your mouse over a link?’ He thought it was just too geeky.”

Melton and Forstall kept it as an option, off by default. The progress bar had lived inside it and needed a new home.

“The room got quiet. Steve and I sat side-by-side in front of the demo machine staring at Safari. Suddenly we turned to each other and said at the same time, ‘In the page address field!’”

“The irony of that invention is that years later I tried to get the whole feature removed. Because even when precision testing showed that Safari loaded pages faster than any other browser, that damn in-your-face progress bar made it seem slower to the user. Its wonderful visibility was killing our reputation.”

2005: The Intel transition

NeXT ported its operating system to Intel in 1993 because it had to survive. In 2005, Apple used that same operating system to move the Mac to Intel. Tevanian ran software through both transitions; he treats the second as the payoff of the first.

John Markoff, interviewing him, remembers Jobs hinting at the switch a year before it happened. Tevanian’s answer is that the product already existed:

“That’s because we already had it working. [...] So one of the things about being in tech, right, is when a product is announced it was already working well before that and so the people who were already working on it knew that it’s coming. Now, he may not have known the exact date but as far as I was concerned, we were working on that product the day I started at Apple in ‘97, and we had software running on all the Macs throughout all those years.”

He did not know which form the move would take, only that it would come:

“For me, having our software running on an Intel processor at some point in time was inevitable, and so it always was about options and how we would execute those options, and so I didn’t know if, like at NeXT we got out of the hardware business, Apple would have to get out of the hardware business, as a really radical change, right, in which case we’d have software to OEM to different people, or we’d be in the hardware business, maybe building premium computers, which Apple’s always been great at, with someone else just doing a super-low-cost computer, right, which would probably be based on an Intel processor.”

The job, as he saw it, was to keep the option cheap:

“I knew that at some point in time Steve was going to say, ‘I need this product.’ Okay. And I knew, starting from scratch, it was a three- to five-year effort, because of everything involved, okay. To port an entire OS and do emulation and everything else, and so I knew I had to have it ready so that when it was time to pull the trigger we could get it out in 9 to 12 months, which is what we ended up doing.”

Serlet, who ran Mac OS X engineering under him, describes the mechanism. Every project had to build for a processor no shipping Mac used:

“We had several kind of points in time where we thought the PowerPC was not moving fast enough from a technology standpoint, and we should move to Intel. One thing that Avie pushed, and I enforced it, is that we always compiled our code for Intel as well as PowerPC, even though we had no Intel machine and so we had some checkers in place in the build system to make sure every single project can be built for Intel and this is years before we did the transition.”

Rubinstein, on the hardware side, had lived the PowerPC problem. On the G5 tower:

“That was a real slog with IBM, who did the processors, G5 processor for-- it was an amazing product.”

Jobs and Paul Otellini signed the Intel deal in February 2005. The developer conference was in May. Serlet:

“We just scrambled to make it happen for the developers conference that was in May, so three months later [...] We wanted also developers to have machines that they can play. But we didn’t want the Mac OS to run on any PC. So there was a delicate balance. So we decided to build machines in secrecy and in fact, we enlisted Simon Patience’s team, which was the CoreOS team, to actually build the machine that we were going to give for developers at the conference. So for a while, the kernel team was actually building, assembling machines in a secret lab in preparation for WWDC.”

The circle of people who knew grew from a dozen in February to several thousand by June, and nobody leaked. The booth staff at WWDC were told an hour before the keynote.

On stage Apple said a year. Inside, Serlet refused to give a date at all:

“We said at that time that it would take about a year to transition and nobody believed it. All the industry thought that it would take much longer. We were hoping it would take less. But we were not sure, because we had a dependency on Intel for some of the new chips coming up. So we were not sure, and still we’re not sure, for several months. So I said, well, we’re going to pretend we need to ship ASAP and we don’t have a schedule, but we just have a punch list of what’s left to do and we’re going to shrink the punch list and that’s what we did. People were upset because they wanted to know the schedule and I said, sorry, I can’t tell you the schedule, I don’t know the schedule, but it’s ASAP, and so we were able to ship in January with the new Intel Macs and so that was six months, not a year and then the rest of the transition was done in less than a year, the rest of the transition of the machines.”

For the year in which both kinds of Mac were on sale, he forced one code base through software update:

“I wanted the transition to be viewed as invisible. So what we did for the rest of that year is we did software updates that had all the improvements that we made to the Intel side. But that also were all those improvements that were not for PowerPC, but we also included that in the PowerPC update and so we had a single code base. We forced the code base to be the same through software update. So by the time January came in with the new Intel machines, they were exactly the same software as the PowerPC. So no one ever found some bug difference between the two.”

Tevanian’s summary of what the customer was supposed to notice:

“One day a Mac had a PowerPC. The next day it had an Intel, right? But it was still a Mac.”

The Intel Mac lasted fifteen years. In June 2020 Apple announced it would move the Mac to its own processors, the same ARM-derived architecture as the iPhone, and by 2023 every Mac shipped on them. The people on tape did not predict that in so many words, but Rubinstein describes the two steps that led there. The first was the iPod’s processor. When PortalPlayer, the merchant-chip supplier for the first iPods, pushed back on Apple:

“We went to Samsung, and we worked out a deal with Samsung where they were going to do multiyear, multi-family processors for us basically to our specifications, and they went and did that, and we got rid of PortalPlayer. [...] They built processors, and I think they built all the processors for Apple until Apple started using their own basically.”

The second was the 2008 purchase of P.A. Semi:

“I don’t think they got the chips, but they got the core design, and they developed on top of it, but for many years we used a Samsung roadmap, a multiyear roadmap and stuff.”

The reasoning goes back to his NeXT years, where a computer took three or four years to build and was out of date when it shipped:

“One of the things I started getting in my head is, we want to do one-year cycles. Now, that leads to a problem in that it takes more than a year to do a chip, right? See, then you have to really have a pipeline of chips you’re working on, and so you have to have enough resources to where you can put in place a roadmap with a pipeline of chips for both processors and control chips and all of that.”

Asked whether that meant making your own chips: “You can buy stuff outside, but if you want to build a real system, you got to do your own chips.” Asked whether that included the microprocessor, in 1990: “No, no, you don’t have to build a microprocessor. You can buy that outside.” Thirty years later Apple built that too.

2005 to 2007: Two iPhones, one shipped

Scott Forstall gave one substantial public interview after leaving Apple in 2012, at CHM in 2017. He tells the origin of the iPhone's interface this way.

The origin begins with Jobs disliking a man at Microsoft. Forstall clarifies that it was not Bill Gates:

"It wasn't Bill, because he was starting to like Bill by this point. It was, Laurene had a friend, so Steve's wife had a friend who was married to a guy who worked at Microsoft. Every time Steve had any social interaction with that guy, he'd come back just pissed off."

One of those social encounters supplied the provocation:

"He came back one time after seeing this guy, and that guy was talking about how Microsoft had solved computing, they were going to do tablet computing, and they were going to do it with pens. And he just shoved it in Steve's face, the way they were going to rule the world with their new tablets with their pens. And Steve came in on Monday and there was a set of expletives, and then it was like, let's show them how it's really done."

Jobs's requirement was no stylus:

"The first thing is, they're idiots, you don't use a stylus. It's cumbersome, you lose it, you're always picking it up and putting it down. We're born with ten styluses. So let's use the ones that we don't have to sell."

That was a tablet project. It became a phone because of what was happening to the iPod. Forstall:

"I think half of our sales at the time were iPods. So we were turning into this consumer electronics company. We were always looking at what was going to take over the iPod space, like was something going to cannibalize music sales and iPod sales, and the one thing that seemed like it might do it would be phones."

Two teams competed. Nitin Ganatra, who ran iOS applications, describes P1, an iPod-derived phone under Fadell, and P2, the OS X-derived phone under Forstall, which Apple internally called Purple. P2 won. Its foundation was the NeXT stack Apple had put into the Mac ten years earlier.

At Macworld in January 2007 Jobs announced the iPhone and dropped the word "Computer" from the company's name. Apple Computer, Inc. became Apple Inc.

2005 to 2007: Twenty-five engineers, a locked room, and a demo every other Monday

The people who built the iPhone's software describe an organisation smaller, more secret and more ritualised than the product suggests.

Ken Kocienda, who wrote the keyboard, on the size of the software team:

"I'm going to make a wild guess just to get you in the right ballpark, four to one engineers to QA. [...] Let's say there's twenty-five engineers, and three, four, or five QA people."

Richard Williamson, who ran the phone's Safari and WebKit work, on why so little QA:

"that meant that individual engineers had to really be on top of the quality of their software. [...] it's kind of unusual in an organization to have so little QA support and to rely on the engineers so much."

Nobody came from outside. Williamson:

"Steve gave us a mandate to, you know, whatever resource you need across the company, go get the people but get the best people. And we were lucky enough to find some really talented folks within Apple. I don't think we hired anybody from outside of the company for quite some time."

Hiring was trust carried over from Safari. Kocienda:

"you take again somebody like Vicki Murley. Why was she brought in onto the iPhone? We worked with her on Safari and WebKit. And we knew she could do the job."

Both men say that is how most of the team arrived, and Williamson adds that it is also why the team was "not diverse at all."

Bas Ording, who designed most of the interface, worked in what had been Apple's usability lab:

"it wasn't used very much at that point anymore. But yeah, there was no windows and stuff, and there was just like a key for that room, and that's where we had that setup."

About eight people from different teams met there weekly or fortnightly, with no team name, alongside their Mac OS X jobs.

Scott Forstall, who ran iPhone software, describes the wider regime: six lockdown areas and a sign reading "the first rule of Purple is you don't talk about Purple." For most of development nobody used the phone as a phone. He decided to be the first, and asked Jobs's permission; Jobs said "sure, go for it, give me one too," and Forstall answered "not yet."

"I slipped an iPhone into my right pocket, I had my flip phone in my left pocket, and I walked out four lockdown areas into the parking lot and I was terrified [...] I drove a different route home."

He traces the apparatus to a teenage summer job at a Navy yard:

"I was always guarded by Marines with attack dogs and semi-automatic weapons, which is exactly how we protected the iPhone later."

Andy Grignon, who ran the radios, on what the rest of Apple saw:

"people would see all of this food being carted in and they thought it was just some luxury lifestyle happening behind these frosted over glass doors. And the funny thing is is when you actually went back there, it stank because the janitor people weren't allowed to clean regularly."

Nitin Ganatra, who ran the applications teams, had a test for anyone asking to be let in:

"'Well, why? What specifically are you running into that you need?' And once you kind of, once you ask a couple of those questions that way, you can really find out does somebody need access or not."

The management system was a demo review with Jobs. Ording:

"usually every two weeks, and they would last about two hours or so. And for a while, we had them every week, and it used to be on Mondays, so the whole weekend, you're kind of like, 'Oh, my demo's not ready! And I don't want to get yelled at on Monday,' so you'd just work the whole weekend to get the stuff better."

They ran just after lunch, which meant no lunch. Ganatra on preparing for one:

"I always had a little bit less sleep the night before the days that I knew I was going to be meeting with Steve, and I was always sort of like, shields are like half up because I'm ready to get chewed out because a demo that I'm going to show isn't going to work [...] 'I don't have to have the answer to every single thing that I might be asked by Steve Jobs in this meeting, but I should really have the answers to 80 percent of the things.'"

Grignon learned the one rule by breaking it. He debugged live in front of Jobs once, and lost his seat at interface reviews for a stretch.

"Steve is very impatient, was very impatient. And the last thing you ever did was sit there and try to fix problems. Like, just cut. Just stop. [...] And move on. You'll get like a little verbal lashing, and that's it."

Kocienda demoed to Jobs six or eight times in total, and the demo was how a decision got reversed:

"There were a couple of times that I changed his mind in a demo. [...] he said he wanted something, and I showed him a demo for something different. And he said, 'Yeah, okay. That's better. That's better than what I was thinking. And you've already got it.' [...] He had no reason to take my word for anything. He didn't really know me. But it was about the work."

Ording says the look and behaviour of iOS were settled before engineering was disclosed:

"the very beginning of the whole, I guess you could call it the iOS look and feel and behaviors was done before engineering was involved, really. Well, of course Scott Forstall was involved. He saw what was going on, but other people didn't really see it until the demo was at a certain point, certain level. And that's what's like, 'Well, this is going to be the thing.' And of course, then there's still lots of discussion about how it should be really built, but the direction was set."

He contrasts it with Mac OS X, where features kept changing while they were being built. The demo that sold the phone to Apple's own leadership in May 2005 was mostly a rehearsed path. Ording, itemising:

"It was a little bit of smoke and mirrors in certain places. [...] Where you could only tap on certain things. There was just one sequence that would work. But some of them were more interactive, so you could, like, like with the music one you could scroll through the whole list and you could pick any song and it would play [...] But I guess the keyboard was probably not working then."

Once engineering took over, Ganatra's rule was that demos ran real software, argued as efficiency:

"if you're working on smoke and mirrors that's time, you're taking time away from working on the actual product. [...] if you can make it so that the thing that you're demoing is the thing that you are going to ship then all of the efforts are aligned as far as making a great demo and making a great product."

As the date approached, control passed to Kim Vorrath's programme office. Williamson:

"the threshold for changes goes up. Every change is a potential bug to be introduced, right? So, you really want to ratchet up the level of what you'll accept. And Kim was very good at this. We used to have BRBs non-stop, bug review boards. And they would be daily or sometimes twice daily moving up towards the release into the launch."

Kocienda, from the engineer's side of the table:

"If you had a bug [fix], you needed to go to this bug review board and defend it. [...] particularly, the closer you get to release, the base assumption is no, we're not going to change this software, convince me that we need to."

Changes that came from Jobs went through the same board. The keynote itself was a test plan. Williamson:

"pretty much for every keynote, there was a script that was mailed out. But it wasn't generally mailed out, so all of the senior managers had the script. And we had to run through the script on our own devices and make sure everything worked and track the bugs that were associated with the script. But Steve did a lot of rehearsals with Phil. [...] they used to call Steve 'the talent.' And you had to make everything perfect for 'the talent.'"

Kocienda, whose keyboard was in the demo, went to Moscone with no idea what the presentation would be. Grignon watched from the third row, having agreed with colleagues the night before that whoever owned each segment would drink a shot when it passed, then realising the radios were in every segment.

"We had never had as good of a run-through in practice, in rehearsal than we ever had at the unveiling, and so, that was a really interesting stroke of luck."

2008: The App Store, and the decision Apple did not plan to make

When the iPhone launched in June 2007, the official position was that third-party developers would build web apps. There would be no native software from outside Apple.

Ganatra says that was also the internal position at first. Developers kept pointing out the difference between Apple's native apps and the web apps available to everyone else:

"Initially it was not different. There were definitely a lot of requests that came between January and even when we shipped, people coming and saying, hey, you know, web technologies aren't going to work as well as what you demoed, so why is that the answer for third parties?"

The team ran into that wall often enough to change course. The good news was that Apple had already built its own apps as if an SDK existed. The less good news was that turning an internal interface into a public promise is an enormous amount of work. Ganatra:

"We went through enough examples like that until we realized that really the right answer here is to release an SDK. And luckily, Scott Forstall, to his credit, he knew what these projects look like and how we were already building the software. He had a good understanding that internally, even though we didn't have a third-party SDK, internally we were developing these things as though we had, using our own internal SDK. So the amount of work that we would have to do, we already have an SDK basically, so we would have to do some sanitizing and cleaning up and getting some interfaces ready to share with the outside world. And Apple takes those interfaces very seriously, so that's an enormous amount of work. You don't just open things up and let people do what they want and then now you have a huge problem later on."

The App Store opened on 10 July 2008 with 500 apps. The interfaces Ganatra describes sanitizing were UIKit, the direct descendant of NeXT's AppKit. Every app on the store was written against a framework whose design dates to 1989.

That is the point at which Apple stopped being a computer company. Forstall had said half of sales were already iPods before the phone shipped. After 2008 the iPhone and the software sold through it became the business, and the Mac became one product line among several.

2011 to 2026: What the third act inherited

Tim Cook became CEO in August 2011. Rubinstein, who had left by then, credits him with a specific contribution from the iPod years: negotiating the flash memory supply.

"Tim Cook did a great job negotiating a deal with them. I mean, he really, we spent a lot of time together over there working on this, and he really convinced them to drop their drawers and give us pricing that was unbelievable."

The company Cook ran for fifteen years rests on three earlier decisions. Its operating systems descend from Mach and NeXTSTEP. Its frameworks descend from AppKit. Its processors descend from ARM, which Apple co-founded in 1990, sold most of its stake in between 1998 and 2000, and returned to when it began designing its own chips.

Apple Silicon, announced in 2020, moved the Mac off Intel and onto the same architecture as the iPhone. NeXT had moved to Intel in 1993 to survive, and Apple repeated the move in 2005. The chip company Apple had once owned 42 percent of eventually undid both transitions.

Ternus was on the hardware team for all of it. Apple's announcement credits him with "multiple new product lines, including iPad and AirPods, as well as many generations of products across iPhone, Mac, and Apple Watch," and says his work on the Mac "has helped the category become more powerful and more popular globally than at any time in its 40-year history."

He takes over a company whose kernel, frameworks, language lineage and chip architecture were all decided by people who thought they were failing.

What today's Apple inherited from NeXT

Mach became the kernel of NeXTSTEP, then Mac OS X, then iOS.

Objective-C, licensed from Stepstone and rebuilt at NeXT, was Apple's primary language until Swift. AppKit and Foundation, NeXT's frameworks, became Cocoa on the Mac and the model for UIKit on the iPhone.

NeXTSTEP's portability to Intel, bought in 1993 to survive, moved the Mac in 2005.

The people: Tevanian ran software until 2006. Serlet ran it until 2011 and hired Federighi, who runs it now. Rubinstein ran hardware and the iPod. Forstall ran iPhone software until 2012. Sina Tamaddon, who ran NeXT Europe, ran Apple's applications division for a decade.

Hoffman's sentence holds all of it: "today's Macintosh success is NeXT."

The people who were there

Everyone quoted above sat for a long recorded interview. Most were recorded at the Computer History Museum. They run about three hours and have a few hundred views.

Each appears with timestamps and transcript links in the companion piece: Apple history in their own words.

Transcripts: Markkula · Sculley, part 1 and part 2 · Amelio · Tevanian, part 2 · Rubinstein, session 1 · Hoffman, part 2 · Forstall, CHM video · NeXT panel with Tevanian, Tribble and Lewin

Misconceptions

Each of these has at least two people on the record, and they do not always agree. The accounts are quoted with the page or timestamp so you can check them; the CHM transcripts are linked once per person below.

John Sculley fired Steve Jobs in 1985.

The two men in the room agree on the mechanism and differ on the word.

Sculley: "So Steve was never actually 'fired' from Apple, but he was demoted from the role of leading the Macintosh division and then he went off on sabbatical." (Sculley, part 1, p.7)

Markkula, who ran the board's investigation and sided with Sculley: "He didn't fire Steve, but he made it so Steve had nothing to do so Steve decided to go off on his own." (Markkula, p.38)

Record: removed as head of the Macintosh division in spring 1985, given a role with no reports, resigned in September 1985.

Common belief: 1985 Apple collapsed as soon as Jobs left.

It grew and stayed profitable for eight years. Apple's revenue went from about $2 billion in 1985 to about $8 billion in 1993, with a profit every year of Sculley's tenure. The decline is 1993 to 1996, under Spindler and then Amelio: Mac OS licensed to clone makers, product lines multiplied, roughly 300 R&D projects running at once, and Copland, the replacement operating system, never shipped. Rubinstein on arriving in 1997: "most of the projects had been actually cancelled before Steve came back because Steve didn't actually come back. I mean, history has been rewritten a bit." (Rubinstein, session 1, p.66)

Misconception: Apple bought NeXT mainly to bring Jobs home.

Three accounts, from the CEO who made the decision, the board member who watched him make it, and the engineer whose software was being bought.

Amelio: "the NeXT operating system was the best choice out there. I would say virtually everybody felt like we shouldn't touch it because of Steve Jobs. Every one of my board members and most of my staff told me forget about it. Nonetheless, I continued to have talks with Steve, and I couldn't find another path we could follow. If I couldn't get Sun, then the third, whatever was third on the list was so far down the list that it really wasn't a viable strategy." (Amelio, p.35)

Markkula: "Gil is the guy that came to the conclusion that we needed to buy an OS [...] The other one we looked at was Be. And Jean-Louis had done a really good job with that, but it wasn't finished." On Jobs: "I don't think it was an issue. If he wanted to come back, fine, if he didn't, fine. I don't think, what we wanted was the OS. And Avie Tevanian." (Markkula, p.42)

Where they diverge: Amelio says the decision to bring Jobs back was the hardest of his life, "I agonized more over the decision as to whether to bring Steve Jobs back more so than I think any other decision I made in my life" (Amelio, p.36), and that ten years on "it was still his company" (p.36). Markkula's account gives Jobs's return almost no weight at all. Both agree the object of the purchase was the operating system.

Myth: The $429 million bought a charismatic founder and little else.

What changed hands: the Mach kernel, NeXTSTEP and the OPENSTEP frameworks, the Objective-C toolchain, WebObjects, and the engineers, including Tevanian, Rubinstein, Serlet and Forstall. Apple's FY2000 10-K books a $375 million charge for in-process research and development. Hoffman, on what the purchase became: "It was NeXT's software that created that Macintosh to the exclusion of the previous Macintosh. People don't remember the previous Macintosh." (Hoffman, part 2, p.22)

Tevanian adds what NeXT did not know it was buying into: "Right after signing the deal to be acquired, so now we're looking at late December, couple days before Christmas, '96, as that quarter is coming to a close for Apple, they had to pre-announce a major loss, and that kind of took us by surprise." (Tevanian, part 2, p.3)

Common shorthand: Jobs returned, introduced iMac, and Apple recovered.

The sequence from 1997 to 2007 has more steps, and the first ones were not his. Rubinstein: the product cuts, the Singapore and Paris closures, and the halving of engineering ran under a McKinsey reorganisation with Fred Anderson as interim CEO, before Jobs held any office (Rubinstein, session 1, p.66 and following). Then, in order: the Microsoft deal in August 1997; the iMac in 1998 and the ARM stock sales that rebuilt the cash position; Mac OS X in 2001, made shippable by Carbon; the iPod in 2001, built around a Toshiba drive shown at the end of an unrelated meeting; the Intel transition in 2005; and the iPhone in 2007, on the same NeXT-derived stack.

Myth: Tim Cook's Apple left the old technical foundations behind.

Cook ran the company from 2011 to 2026 on NeXT's operating system and frameworks and ARM's chip architecture. Apple Silicon, from 2020, put the Mac on the same architecture as the iPhone, reversing the Intel move of 2005. Cook becomes executive chairman on 1 September 2026; John Ternus, head of hardware engineering, becomes CEO.

The tidy version: Jobs made NeXT a route back into Apple.

Larry Ellison said so at Jobs's memorial. Two people who were inside NeXT at the time say otherwise, on the record.

Rubinstein, asked whether Jobs intended to return: "Not coming back, yeah. No, no, not coming back. [...] And that bullshit that Larry spun, Ellison spun at his eulogy, at his funeral and stuff was nonsense." He adds: "I know Laurene didn't want him to go back, right, because it meant he would disappear." (Rubinstein, session 1, p.64)

Tevanian, on the paperwork NeXT had ready before Apple called: "if you look at the S1 that we were ready to file when we were going to go public he was going to basically not be the CEO anymore and we would have had an office of the president." (NeXT panel, 48:10) The company was preparing to move Jobs out of the CEO role so he could spend more time at Pixar, weeks before the acquisition talks.

Rubinstein concedes the limit of his own evidence: "Anything is possible, but we were pretty close at the time." (p.64)

Misconception: ARM money funded the NeXT purchase.

Sculley: "I give Gil Amelio credit for two things, one was he sold the 47 percent that Apple owned of ARM for about 800 million dollars, or so I'm told, and that 800-million dollars was crucial to keeping Apple alive, and he brought Steve Jobs back." (Sculley, part 2, p.6)

Apple's filings disagree on the stake, the timing and the amount. The FY1999 10-K puts the holding at 42.3 percent in September 1997 and dates the first sale to October 1998, fifteen months after Amelio left. That month Apple sold 2.9 million shares, taking the stake to 19.7 percent; fiscal 1999 sales brought about $245 million and a gain of about $230 million; a further 5.1 million shares went in the first quarter of fiscal 2000 for about $136 million.

Where they agree: the stake mattered to the recovery. Where they do not: who sold it, when, and for how much. The cash arrived after the NeXT acquisition and during the turnaround, not before, and not under Amelio.

CHM has been interviewing the NeXT and Apple engineers in three-hour sessions. Most of these have just a few hundred views, but they are packed with interesting stories and knowledge.

Steve Jobs in his own words

There are a lot of great Steve Jobs interviews convering Apple founding story and early stories. These four are the ones I keep going back to. But the details from the people who worked alongside him are what I find more interesting, and that is the rest of this post.

Steve Jobs, Santa Clara Valley Historical Association, 1994

29 minutes, interviewed by John McLaughlin, filmed at NeXT in Redwood City.

Recorded three years before he went back to Apple. He covers the founding of Apple, the Macintosh, being pushed out, and building NeXT.

Steve Jobs oral history interview, 20 April 1995

1 hour 23, interviewed by Daniel Morrow of the Computerworld Smithsonian Awards program.

This is the full unabridged recording. The Smithsonian hosts the transcript, and the Steve Jobs Archive has published it as “Pursue different paths” in 16 chapters.

Steve Jobs, The Lost Interview, 1995

1 hour 6, interviewed by Robert Cringely for the documentary Triumph of the Nerds.

Only about ten minutes of this aired. The master tape was considered lost until a VHS copy turned up in the director’s garage after Jobs died.

Steve Jobs at MIT Sloan, 1992

1 hour 12.

An hour on how NeXT actually made money, management practices, future of NeXTSTEP developers, OOP.

Before Jobs’s exile

The people who left Apple with Jobs in 1985 were Macintosh people.

Andy Hertzfeld

Second software engineer on the Macintosh team. 6 hours 20 across two parts, recorded September and October 2025.

Part 1 covers Hertzfeld joining the Apple II team in 1979, porting QuickDraw from Lisa to Mac, building much of the Toolbox, and his conflict with manager Bob Belleville, which Hertzfeld says contributed to his departure after the Mac launch. Part 2 covers Radius with Burrell Smith in 1986, Frox with Hartmut Esslinger, co-founding General Magic in 1990 with Bill Atkinson and Marc Porat, Eazel in 1999, then eight years at Google working on Photos, News, Plus and Glass.

CHM files part 2 under “Hartzfeld,” so a title search misses it.

Mike Markkula

Apple employee number three, wrote the Apple II business plan, on the board until 1997. 2 hours 35. Transcript.

On what Apple was buying in 1996, Markkula says: “what we wanted was the OS. And Avie Tevanian.” On Jobs returning with it: “I don’t think it was an issue. If he wanted to come back, fine, if he didn’t, fine.”

On the earlier IBM approach, Markkula says: “Lou Gerstner. It was his idea to buy Apple for a song.” On how the OS decision was made: “Gil is the guy that came to the conclusion that we needed to buy an OS. The other one we looked at was Be. And Jean-Louis had done a really good job with that, but it wasn’t finished.”

Skim for those passages, around pages 40 to 42 of the transcript.

Larry Tesler

Came to Apple from Xerox PARC and later founded Apple’s Advanced Technology Group. 8 hours 15 across three parts. Part 2 covers the PARC demos to Jobs; part 3 covers ATG and the Apple Fellows.

In part 3, Tesler explains what the Apple Fellows recommended around 1995: they debated porting the Mac OS to Intel, rejected it, and advised buying an outside OS, “either the NeXT operating system or the Be operating system.”

Bill Atkinson and Andy Hertzfeld on MacPaint

Atkinson describes Jobs recruiting him off an unfinished neuroscience PhD when Apple had 30 employees. Includes a live MacPaint 1.0 demo. 1 hour 20. Transcript.

Bruce Horn

Horn was at PARC from age 14 and was there during Jobs’ 1979 visit. He created the Resource Manager, resource forks and file Types and Creators, which is why the Mac did not need filename extensions. He later co-founded a company built on NeXT’s WebObjects. Wrote the Macintosh Finder.

Steve Wozniak

3 hours 4, University of Washington, 2006. A guest lecture in Ed Lazowska’s.

NeXT, 1985 to 1996

Five people left Apple with Jobs: Rich Page, George Crow, Susan Barnes, Bud Tribble and Dan’l Lewin.

Bud Tribble and Steve Jobs at a NeXT retreat, 1986

Tribble, NeXT’s founding software lead, says he was told to build a word processor with no feature spec and therefore cannot commit to the launch date. Jobs pushes on the date. Tribble holds.

Bud Tribble

1 hour 23, University of Washington, 2006. From the same course as the Wozniak lecture, it also circulates only as two reuploads without descriptions. I identified it by matching the guest roster: Lazowska, Kate Deibel, Mike Koss and Burton Smith are all UW or Seattle people.

Rich Page

NeXT co-founder and head of hardware. 5 hours 50 across two parts. Part 1 covers Page testing memory chips at Fairchild Semiconductor in 1972, moving to HP, then joining Apple in 1978 to work on the Lisa. Page says he convinced the Lisa team to drop a custom microcoded bit-sliced processor in favour of the Motorola 68000. He built large-screen, colour and portable Mac prototypes and became an Apple Fellow alongside Atkinson.

Part 2 covers NeXT manufacturing: the automated factory, the difficulty of making the magnesium cube, supplier relationships with Motorola and Canon, the second-generation NeXTstation, and the NeXT RISC Workstation, which never shipped. Page left in 1992, by which point every co-founder except Jobs had gone. He then founded Sierra Research and Technology, which made 10/100 Ethernet chips and was sold to TDK in 2000.

It is the only detailed account here of NeXT hardware.

Dan’l Lewin

NeXT co-founder, sales and marketing. 2 hours 43. Transcript. Lewin was selling Sony tape recorders and met Jobs in a meeting about Sony’s 3.5-inch floppy drive, which Apple went on to choose for the Macintosh. Jobs recruited him. He built Apple’s education sales and marketing, then followed Jobs to NeXT to sell the cube into universities. After NeXT: GO Corporation, Aurigin, Kidsoft, then Microsoft from 2001. He now runs the Computer History Museum.

Avie Tevanian, part 1

Author of the Mach microkernel, later Apple’s SVP of Software. 2 hours 31. Tevanian started Mach as a graduate student under Rick Rashid at Carnegie Mellon. NeXT engineers decided they wanted it after seeing it presented at a UNIX conference in 1986, and Jobs relayed the interest at a dinner in Palo Alto. Tevanian turned down an offer from Microsoft and joined NeXT. He ran the core OS team for a year, then the whole OS group, then became VP of Software after Tribble left.

Bertrand Serlet

AppKit engineer at NeXT, later Apple’s EVP of Software. 2 hours 52. Serlet was at Xerox PARC in 1984, then followed Jean-Marie Hullot from the French research institute INRIA to NeXT in 1989. He worked on the AppKit team and shipped OpenStep, then led the Foundation framework, then Enterprise Objects Framework and WebObjects. At Apple he led Rhapsody, which became Mac OS X, and took over software after Tevanian. He left in 2011 after recruiting Craig Federighi to replace him.

Blaine Garst

Tevanian hired Garst in 1990 to manage the Mach kernel team. With Serlet he added protocols to Objective-C. He implemented Distributed Objects, added reference counting to memory management, and wrote many of the original Foundation classes alongside Serlet and Ali Ozer. In 1996 he built the Objective-C to Java bridge that let WebObjects applications be written in Java.

Steve Naroff

Ran NeXT’s development tools. Naroff was at Stepstone, which sold Objective-C, porting the compiler to HP-UX and Apollo. He became the main engineer fixing the language for NeXT’s needs, felt Stepstone was not resourcing that work, and joined NeXT in 1988. There he integrated Objective-C into GCC and added categories to support Interface Builder. Later at Apple: director of Java technologies and developer tools, then Xcode, then Objective-C 2.0, then the LLVM/Clang front end from 2007 to 2010.

Naroff also co-wrote the language history with Brad Cox, who created Objective-C, and CHM curator Hansen Hsu: “The origins of Objective-C at PPI/Stepstone and its evolution at NeXT”, open access.

Glenn Reid

NeXT engineer, later built iMovie. Reid got into Adobe’s font group after writing Adobe a letter in PostScript. He joined NeXT in 1990 working on email and fax, left in 1991 to found RightBrain Software and ship a NeXT desktop publishing app. In 1998 Jobs hired him to build iMovie 1.0, which Reid says he built in nine months with two other engineers. He then led iPhoto through version 4 and was Director of Engineering for Consumer Applications until 2003.

Joanna Hoffman

Original Mac team, then NeXT. Start with part 2. Hoffman wrote the Macintosh business plan and followed Jobs to NeXT after 1985. She left NeXT after nine months. On the pitch she kept hearing: “Eighteen months we’re going to have this incredible product,” and then “not again!” about the focus on an unrealistic storage device. Hoffman thought the NeXT vision was too small for Jobs.

Hoffman’s assessment of how it turned out: “I keep saying the first Macintosh’s success was really Windows. But today’s Macintosh success is NeXT. You know? No question.”

Parts 1 and 3 cover Soviet Armenia and Poland, emigrating to Buffalo at 13, MIT and PARC, then what working for Jobs was like.

The acquisition and the return, 1996 to 1998

Avie Tevanian, part 2

Tevanian says NeXT went from first call to signed acquisition in under a month, and that days later Apple pre-announced a large loss, which took them by surprise. He covers the bake-off against Be from NeXT’s side, working under Amelio and then Jobs, pushing TCP/IP and open standards over Apple’s proprietary technology, the Intel transition, sunsetting Newton, Mac OS 9 and WebObjects, and his Microsoft antitrust testimony. He retired in 2006.

Tevanian also describes Fred Anderson phoning Ellen Hancock to fire her right after firing Amelio, and Hancock initially thinking the call was to make her CEO.

Jon Rubinstein

Ran Apple hardware from 1997 and later the iPod division.

Rubinstein started on 3 February 1997, the day the NeXT deal closed. He describes Apple as worth around $2 billion with one quarter of cash left, and says they hid him for three days because he was there to replace five division managers.

On Larry Ellison’s claim that Jobs had planned to retake Apple, Rubinstein says: “That is certainly not what I saw. Laurene didn’t want him to go back.”

The iPod material is in session 2. Rubinstein says IBM’s Microdrive group “laughed at me” when he asked for five gigabytes at his price. He found the Toshiba 1.8-inch drive at the end of a Japan roadmap meeting about something else, called Jobs in Tokyo to ask for a $10 million check, then called Fred Anderson to check it would clear. Phil Schiller supplied the scroll wheel, from an old Bang and Olufsen phone, and Apple patented it and licensed it back to B&O. The first iPod was 20 to 25 people in eleven months, and the launch moved a week for Rubinstein’s wedding.

Rubinstein also says Apple’s multi-touch origin story is “all nonsense,” and blames Verizon for killing the plan to put cellular in every laptop.

Regis McKenna

Marketed the Apple II and knew Jobs for 35 years. Part 5 of eight, 2 hours 3, scoped to 1985 through 1999.

McKenna reads from a dated notebook. His entry for 6 January 1997, after Jobs met Amelio: “they don’t appear to want help. They didn’t offer him a job.” For 9 August 1997, when Jobs phoned him: “What do you think about my new board?”

McKenna on Amelio’s executive hires: they “all wanted to be miniature Steve Jobs,” which he likens to the Hell’s Angels dressing up in a tuxedo. On Michael Spindler: “He could stand at a board and draw out a future strategy, and it would overwhelm you. He fell short in how you implement it.” McKenna turned Jobs down twice on rejoining.

Gil Amelio

Apple CEO from February 1996 to July 1997, and the person who bought NeXT. Transcript, 42 pages, interviewed by Jeff Katz and George Scalise in 2012.

Amelio on the state of the company when he arrived: “there were 300 R&D projects—300. I could make a case that maybe we needed 20 of those, and that might even been too big a number, but 300 was ridiculous.” He got it down to about 50. On manufacturing: “We were getting back—some months—as much as 10 percent of the computers we shipped.”

He went to Sun first. Scott McNealy wanted to buy Apple, and the question was whether the Mac interface could sit on top of Solaris. Amelio: “At the end of the day, Scott McNealy’s board turned us down. At that time, John Doerr was on their board, and I think he was, perhaps, most vocal in saying, ‘No, we shouldn’t make this connection.’”

Then Be, and in Amelio’s telling Gassee came to Apple rather than the reverse: “Jean-Louis Gassee approached us about the Be operating system, and I’m kind of glad he did because it gave us another contender in the stack.”

On choosing NeXT, Amelio says: “the NeXt operating system was the best choice out there. I would say virtually everybody felt like we shouldn’t touch it because of Steve Jobs. Every one of my board members and most of my staff told me forget about it.” He kept talking to Jobs anyway, because “if I couldn’t get Sun, then the third—whatever was third on the list was so far down the list that it really wasn’t a viable strategy.”

And on the decision that ended his own tenure: “I agonized more over the decision as to whether to bring Steve Jobs back more so than I think any other decision I made in my life.” His reasoning was that ten years after leaving, “it was still his company. His DNA was so much in that company.” Amelio on Jobs: “He didn’t understand technology, but he did understand user interface better than anyone on the planet.”

Amelio makes one claim I have not seen corroborated anywhere: that the iMac did not originate with Jobs. He says he killed the Performa, needed a new all-in-one to replace it, gave Ellen Hancock the job, and was “at least six months into that thing—maybe longer—when I ultimately left.”

There is also a 1998 audio interview with him by Paul Schindler, and his book, On the Firing Line.

Larry Tesler, 2013 session

Video listed above

Tevanian asked Tesler to go into the Advanced Technology Group, which Tesler had founded, and work out what was worth keeping. Tesler: “So I was sent back there to kill the organization I had founded, but I’m glad he did it.” Inside, Tesler found a project he describes as essentially Microsoft Surface, “15 years ahead of its time.”

John Sculley

Apple CEO 1983 to 1993, interviewed by David Greelish in 2011 and donated to CHM. Transcripts: part 1, part 2.

Sculley’s argument for not licensing the Mac OS: the models showed that it required about 25 percent market share, a higher OS price than Windows, and layoffs covering 70 to 80 percent of the workforce, while Microsoft charged OEMs “about $11 a copy.” He says that when he left, Apple was “the number one selling personal computer in the world” with an 8.3 percent worldwide share.

His most-quoted recorded moment is eight minutes at a Forbes conference in 2013, on May 1985.

Footage from the time

Macworld Boston, 6 August 1997. Jobs on stage, Gates by satellite.

The new board, then the Microsoft deal: cross-licensing, Office committed to the Mac, $150 million in non-voting stock. Gates appears on the screen behind Jobs and the room boos. Jobs later told Walter Isaacson it was “my worst and stupidest staging mistake of my life. It made me look small, and Apple look small.”

The 1997 WWDC fireside chat, 1 hour 11. Jobs taking developer questions while holding no title at Apple.

The first question is about OpenDoc, and Jobs says he is in favour of putting a bullet in its head. He also gives the answer about starting with the customer experience and working backwards to the technology.

The internal Think Different talk, 23 September 1997. 14 minutes, Jobs to Apple employees.

“Marketing is about values.” He also gives the media plan: the spot breaks that Sunday on ABC around the network premiere of Toy Story.

Also: Macworld January 1997, Amelio’s last keynote, a week after the acquisition was announced, and the first time Jobs and Wozniak were on an Apple stage together since 1984. And the WWDC 1997 opening keynote, where Amelio is still CEO, Tevanian and Rubinstein already run software and hardware, and a young Scott Forstall does a demo.

Mac OS X

Bas Ording

Designed the Aqua interface and later the iPhone UI. 2 hours 14. Transcript.

Jobs called Ording at home in 1998 to recruit him against an offer from MetaCreations. Ording describes the first multitouch rig as a FingerWorks touchpad with a paper overlay, projected onto a tablet and cabled to a Mac.

He says rubber-band scrolling started as a bug: he thought his code had stopped running when a list would not move, realised it was pinned at the top, and built the spring-back from that.

Nitin Ganatra on the Debug podcast

Audio. CHM has the transcripts of episodes 39 to 41.

Ganatra came up through Apple developer support and watched Copland fail from the support queue. He explains Carbon: Adobe and Microsoft would not rewrite for Cocoa, so Apple built a Mac Toolbox emulation layer. Tevanian says in his part 2 that there was never a formal date to turn Carbon off.

The Aqua unveiling, Macworld San Francisco, 5 January 2000. Also the keynote where Jobs drops the “i” from iCEO.

John Siracusa’s Mac OS X reviews, from the 1999 developer previews through 10.0. Text. This is where the NeXTSTEP-to-consumer-OS transition is documented in detail: Yellow Box and Blue Box, Carbon against Cocoa, Quartz replacing Display PostScript, the Finder. The pre-2012 URLs are legacy paths and need checking.

The iPhone

Nitin Ganatra

Director of iOS applications. 5 hours 55 across two parts. Transcripts: part 1, part 2.

Forstall pulled Ganatra off the OS X Mail team in late 2005, and he brought two Mail engineers plus Scott Herz and Alex Aybes with him. Part 1 covers the internal competition: P1, an iPod-derived phone under Tony Fadell, against P2, an OS X-derived phone under Forstall. P2 won. It also covers the web-versus-native fight over the built-in apps.

Part 2 covers why copy and paste and MMS were cut from version one, AT&T’s technical approval process, and the work of sanitising UIKit for public release.

Andy Grignon

Ran the iPhone radios. 3 hours 9. Transcript.

Grignon was an Apple ATG intern on QuickTime Conferencing, then at Pixo, which Apple contracted to produce the OS for the original iPod, then back at Apple for iChat and Dashboard. He describes himself as “the software mole from Forstall’s organization into Fadell’s.” He says he blew a live iChat demo in front of Jobs when the audio cut, and was excluded from UI review meetings afterwards. He once drove to Jobs’ house to debug the radio on Jobs’ personal iPhone. After Apple he followed Rubinstein to Palm to lead webOS.

Ken Kocienda and Richard Williamson

Built Safari and the iPhone keyboard. 6 hours 32 across two parts. Transcript, part 1.

In June 2001 Forstall asked Don Melton to start a browser. Melton hired Kocienda and Williamson, and they built on Konqueror’s KHTML rather than Mozilla. Williamson gives the numbers: Mozilla was about a million and a half lines of code, KHTML about 150,000. “It was three guys and we figured 50,000 lines of code each.” Williamson says he got KHTML running on a Mac in a couple of days by convincing the code it was still on Linux.

Kocienda won an internal contest to design the iPhone keyboard. He describes the winning prototype as not one key per letter: keys were ganged together and software guessed the word from the letter buckets.

Part 2 covers secrecy, working with AT&T, the native-versus-web SDK decision, copy and paste in iOS 3, and Williamson on the Apple Maps launch and his departure from Apple afterwards.

Scott Forstall

Led iPhone software. 1 hour 3, interviewed by John Markoff in 2017.

Forstall on the origin of the multitouch push: Jobs came in on a Monday after a Microsoft executive had bragged to him about pen tablets, and “there was a set of expletives and then it was, ‘let’s show them how it’s really done!’”

The full event with the engineer panel is here; Forstall starts at 1:01:51.

Tony Fadell

Ran the iPod division and led hardware on the first three iPhones. 1 hour 18, interviewed by John Markoff in 2017.

For longer interviews with Fadell: Lenny’s Podcast, Lex Fridman, or Tim Ferriss

Panels

Steve Jobs in Exile, May 2026. 1 hour 12, with Dan’l Lewin, Bud Tribble and Avie Tevanian on stage and Rich Page contributing by recorded video.

Three NeXT founders in one room. It is filed under the title of the moderator’s book, which is why it is hard to find.

Apple at 50, 1 hour 38, with Sculley, Chris Espinosa and Ron Wayne on stage, Rubinstein and Tevanian by video. Steve Jobs Stories, 1 hour 34, which includes Sculley and Hertzfeld. The Churchill Club panel from five weeks after Jobs died, with Atkinson, Gassee, Hertzfeld, McKenna and Tesler.

Other tech history material?

If you know more of these, please share in the comments.

Some of my other favorites: Masters of Doom, David Kushner’s book on id Software; UNIX: A History and a Memoir, Brian Kernighan’s account of Bell Labs; and The Making of Prince of Persia, which is Jordan Mechner’s own journals from 1985 to 1993, published exactly as he wrote them at the time.

Level 3 spec-driven development treats the spec as the source and regenerates the code from it. Nope, bad idea. The spec is scaffolding — critical, then removed.

What spec-driven development is

Spec-driven development means writing the specification before an agent writes code, and treating that recorded intent as the durable artifact rather than the prompt or the output.

Tessl's definition is the cleanest I have found: "the structured counterpart to vibe coding: before the agent writes code, you produce a specification, requirements, design, and a task breakdown, as an editable, human-readable artifact, and the agent builds against it." That separates it from better prompts, which are disposable, and from vibe coding, which keeps no record of intent at all.

The tools often split by role. GitHub Spec Kit defines the canonical four-phase workflow: Specify, Plan, Tasks, Implement, each producing its own markdown file. AWS Kiro builds that workflow into an IDE. Tessl makes the strongest version of the claim, marking generated code do-not-edit. Claude Code and GitHub Copilot sit on the consumption side, reading whatever conventions file you hand them.

There are three levels.

1. Spec-first means you write a spec and use it for the task in front of you.

2. Spec-anchored means you keep it for maintenance.

3. In spec-as-source, humans edit only the spec and never the code.

Birgitta Böckeler, who wrote that taxonomy after running all three tools herself, is not sold on any of them. Her objection is the one a skeptical reader arrives with: "I'd rather review code than all these markdown files." She ran Spec Kit and got back a pile of repetitive, verbose documents, and asks whether the whole thing is a Verschlimmbesserung, the German word for making something worse while trying to make it better.

She is right about what she reviewed. Our answer is that not all of the markdown is written for a person. The pitch a colleague reads is 62 lines. The implementation file an agent reads is 978.

Why prompt-first workflows break down

Why not just prompt? This exists because prompting alone stopped scaling. A prompt-first session builds a shared understanding that erodes as the session runs and vanishes when it ends. Nothing carries to the next one.

There's some data on the impact of AI that seems pretty accurate according to my own experience. Copy-pasted code rises from 8.3% to 12.3% while refactoring fell from 25% to under 10% across 2021 to 2024, in an analysis of 211 million changed lines.

That debt compounds differently now, because agents read the repo to decide what to write next. A mediocre pattern merged today is the house style tomorrow.

Scaffold-driven development: our own version of SDD approach

The big idea in spec-driven development is that the spec outlives the code, and now you code in spec. You round-trip through it, and when requirements change you edit the spec rather than the implementation. Generated code do-not-edit for exactly that reason, with requirements becoming the long-lived thing at the centre.

Not every tool goes that far. Kiro deletes the spec once the feature is built. That one is closer to us, and it is also the version I like least: it solves the maintenance bill by throwing away the reasoning.

What worked out the best for us at GrowthX was the opposite. Write the spec with that much rigor and then trash it (sort of).

At merge it stops being authoritative. The tests carry the contract. Whoever finished the work moves the folder into docs/plans/executed/, and nothing in there governs anything again. That directory in GrowthOS, our core product, currently holds 42 plans and 645 markdown files, about 1.46 million words. Another 48 plans are live.

The word for that is scaffolding. Engineered, load-bearing, safety-critical, and designed from day one to be "taken down" when the construction is finished.

Both halves matter.

A spec you do not take seriously is not scaffolding, it is a sketch.

A scaffold that never comes down is a second copy of the truth, with a maintenance bill and no test protecting it.

It loses its authority and keeps its information. That distinction is what stops this being waste, and I come back to it below.

We are spec-first, deliberately, and we reject the spec-as-source part.

Why we review the spec, not the code diff

We are about 20 engineers and just in our core platform we are merging some 397 pull requests a month. 86% of them carry almost no human review comments (we do have a ridiculous amount of agent reviews, though).

That number looks like a rubber stamp, and it would be one if the diff were where our judgment lived. A study of 400 repeat reviewers found that approval rates rise and inline comments fall 22% as agent PRs pile up. Latency more than triples.

So we moved the examination upstream. The spec defines scope and breakage risk before a line exists. Tests and CI still check behavior. What the spec review replaces is the fiction that a human skimming a diff was doing design review.

Example of what our specs contain

Three files, and their proportions

The "Find Opportunities, from anywhere" plan shipped as two PRs #2766 and #2872. Three files.

pitch.md, 62 lines. The problem and the idea, followed by a Non-goals section. Its main move is collapsing four fuzzy agent verbs (Develop, Refresh demand, Find new ideas, Research opportunities) into two clear stages. Non-goals is where the tradeoffs get made. Deciding what you will not build is the cheapest scope cut available.

ux-ui.md, 406 lines. ASCII wireframes for seven surfaces, each drawn today then after. A naming map of every string that changes. Empty and edge states. Copy drafted in the spec instead of improvised during implementation.

implementation.md, 978 lines. Three slices. We labeled slice two the risky one and isolated it on purpose. Real file paths and migrations. Acceptance criteria and architectural constraints. The reasoning sits between the code rather than after it. Sometimes the implementation is a folder with multiple slices.

Markdown in the repo, not a doc tool

The spec lives in docs/plans/, next to the code it describes. The agent reads it directly. It versions with the code. It diffs in review like anything else. While the work is live, one file is the source of truth for humans and agents both.

Not every change earns a spec

Tiny changes can stay in the conversation and never become a file. Small to medium sometimes can fit in a single file, but high-impact work gets a directory with an index and numbered phases.

We tend to spec the work in phases, but as functional slices. Same approach described here: Shape Up.

How a coding agent consumes a spec

A structured spec beats raw prompting for a mechanical reason. It front-loads decisions into the context window instead of hoping the model recalls them mid-session. Token limits then shape how you modularize: we sliced that 978-line document into three so a working session carries one slice, not the whole plan.

The part worth copying is the grounding facts section our implementation docs open with. Before planning starts, every claim gets verified against real code and cited to file and line, with a date on it. The agent builds on checked assumptions rather than recalled ones.

It also draws a boundary. The agent records what it confirmed statically and names what a human has to verify against a running system. An agent that says "I confirmed the webhook config in code" is useful. An agent that says "I could not confirm the retry behavior without a live run" is more useful, because it tells you exactly where to spend your attention.

What a human review on a spec looks like

During the creation of the spec, we run extensive rounds of Codex adversarial reviews to catch things Claude does not catch.

In this example, before any code existed, Codex ran three review passes against the spec.

Pass two ran after the spec had changed and found ten more. The spec treated a status field as a free string. The system validates it as an enum. Pass three verified the fixes and caught the bugs the fixes themselves had introduced.

Twenty findings later, and still zero lines of code, every fix is cited inline in the spec, next to the paragraph it changed, so the document carries its own review history.

The third pass to catch the last defects introduced by the fixes.

What a retired spec is still good for

Losing authority, keeping information

An executed spec keeps its information. It no longer says what the code should do. It says why: what we considered, what we rejected, and what it cost to find out.

This is the part Kiro throws away and Tessl pays forever to keep. Dropping the authority and keeping the reasoning was the whole trick for us.

We often keep a decision record in our repo with a section titled "Rejected, and why (the part that stops the argument recurring)." Every rejection in it names the prototype that killed it. This is basically an ADR (architecture decision record).

Design docs were a graveyard when only humans read them.

People do not reopen old documents. Agents do.

Retrieval is cheap for them, and they have no ego about reading history before proposing something we already rejected twice.

That is what makes writing specs this carefully worth paying for now, and it has nothing to do with code generation or re-generation after execution, as level three of "spec dev" suggests.

Drift, and why it is a smaller problem for us

Two things catch drift while a plan is live. We re-run the adversarial review against the real PR with the final code diff, slice by slice. And there is a hands-on QA pass before a plan retires.

After merge our spec has retired, so there is no round-tripping obligation and no living document to re-sync. Test failures catch drift in covered behavior. Uncovered behavior can still drift quietly, and does.

When requirements change, we open new work with fresh grounding facts rather than editing an authoritative document.

Spec-driven development next to TDD and BDD

TDD and BDD constrain behavior at the test layer and stay live for the life of the code. Dan North framed BDD as "outside-in", starting from business outcomes and drilling into features. A spec constrains intent before the work exists at all, and in our model it retires at merge.

They constrain different layers. The clearest framing I have read is that SDD anchors the work in a specification, BDD protects the macro behavior, and TDD improves the micro design. In our version that is literal: at merge, the behavioral contract moves into tests and the spec goes to executed/.

We also have our own way to write QA tests with Chrome MCP, inspired by Gherkin, the language behind BDD's famous Cucumber framework. Something I might cover another day.

If you run a different arrangement, especially spec-anchored at volume, I'd love to know your experience in practice.

How is spec-driven development different from writing better prompts?

A prompt is disposable and lives in one session. A spec is a versioned artifact in the repo that people and agents build against across many sessions. The prompt-first failure is that the shared understanding vanishes when the session ends.

What goes inside a spec?

At minimum: the problem, explicit non-goals, acceptance criteria and architectural constraints. Ours add UX wireframes with drafted copy and a naming map, sliced implementation steps with real file paths and migrations, and a dated grounding-facts section citing code to file and line.

What are the three maturity levels, and where should a team start?

Spec-first means write it and use it for the task. Spec-anchored means keep it for maintenance. In spec-as-source, humans edit only the spec. Start at spec-first. It captures most of the review-quality benefit with none of the round-tripping obligation. We stayed there on purpose.

How do you detect and recover when generated code drifts from the spec?

While the plan is live: re-run adversarial review against the real diff per slice and before merge, plus a hands-on QA pass against a real workspace. After merge our spec has retired, so the test suite catches drift in the behavior it covers.

How does code review change when an agent writes most of the code?

Design judgment moves to the spec, before code exists. Diff review narrows to checking that the implementation matches an already-argued plan. Most of our merged PRs carry almost no comments because the findings that mattered landed on a markdown file weeks earlier.

Notes

1. In GitHub Spec Kit: Specify writes spec.md, Plan writes plan.md plus supporting research and data-model files, Tasks writes tasks.md, and Implement executes those tasks in dependency order. Spec Kit was announced on 2 September 2025, based on John Lam's research. AWS Kiro's equivalents are requirements.md in EARS notation, design.md, and tasks.md; it reached general availability in November 2025. Claude Code reads CLAUDE.md and can import AGENTS.md; GitHub Copilot reads .github/copilot-instructions.md and supports AGENTS.md directly.

2. GitClear's analysis of 211 million changed lines. Two adjacent findings point the same way: METR's randomized trial found experienced developers took 19% longer with AI while believing they were faster, and a 2026 code-ownership study found AI-assisted participants answered questions about their own code correctly 87.5% of the time, against 100% without.

3. Lethbridge, Singer and Forward, where 44% somewhat agreed and 24% strongly agreed that documentation is always outdated relative to the system.

The standard definition: an ideal customer profile (ICP) is an account-level description of the companies you are best at winning, serving, retaining and expanding. It is a short list of firmographic and technographic criteria, plus explicit disqualifiers, scored and tiered in the CRM.

An ICP describes companies rather than people. A buyer persona is the person inside the account, TAM/SAM/SOM are market-size estimates in dollars, and an ABM target list is the named accounts you are working right now.

That definition is correct, and it produces some pretty useless docs. Sort your customers by industry and headcount and you get a profile that describes who they are while saying nothing about why they bought.

We wrote ours for the GrowthOS launch and now 2 months in revisited focused on the way round, starting from the job. Christensen: “The job, not the customer, is the fundamental unit of analysis.” Bob Moesta’s version changes what you do on a Tuesday: demand starts at a struggling moment and “it’s not an imagined customer or persona, it’s real buyers.”

So the first question is not what kind of company buys this. It is which organizations hit the struggling moment this product resolves, what progress they are trying to make, and what they are hiring today instead. Firmographics come back as filters on that answer, never as the explanation for it.

The other half of a real ICP is that it is enforceable: an SDR can disqualify an account against it, RevOps can score and route on it, and it gets reviewed monthly against closed-won data.

Revisiting our ICP

We wrote our ICP doc before GrowthOS launched in June 2026. It was a hypothesis with a template around it: which companies the platform would be structurally best at winning, extrapolated from work we had been doing for clients by hand. The first paid clients of the platform have since converted onto annual plans, which is the first evidence in the exercise that isn’t a guess, so we went back to the doc.

Before touching it, I created this study guide for myself, but I’m sharing it here since it might be a helpful resource.

A caveat on the numbers: they come from different years, samples and definitions, so several of them disagree. Use them as shape, not as truth.

Audio version

Here’s a computer generated audio version of this guide adapted for listening

What an ICP is, and what it isn’t (TAM, target lists, personas)

RevOps Co-op defines an ICP through four operational questions: who will buy, who will purchase smoothly and on time, who can be implemented and serviced, and who will renew, refer and expand.

Only the first is a sales question. An ICP built from “who signs” systematically overweights accounts that close and churn, which is why retention and expansion data belong in the derivation rather than in a CS dashboard.

The definitions worth having in front of you. HubSpot grounds it in data: an ICP is “based on company level firmographic and technographic data points (like employee size, technologies used, industry, and more).” First Round Review adds the temporal part: “a detailed description of the perfect customer (in the case of B2B, that means company)” and “a living definition that needs to be revisited frequently.” SaaStr raises the stakes: “Your ICP strategy is the basis for your entire company’s strategy.”

Four layers, four different questions:

• TAM/SAM/SOM works at the market aggregate. How large is the opportunity? The output is a dollar figure.

• The ICP works at the account level. Which types of companies fit best? The output is attribute criteria and a scoring model.

• An ABM target account list works at the named-account level. Which specific companies are we pursuing now? The output is a prioritized list of names.

• A buyer persona works at the individual level. Who inside those companies do we engage, and how? The output is a role, goal and pain profile.

The sequence runs: size the market, define ICP attributes, score the addressable universe, apply capacity and territory filters to get the named list, then map the buying committee with personas.

Clearbit draws the market-sizing line cleanly: “TAM, SAM, and SOM identify market potential and are represented by a dollar value. Your ICP takes that a step further and serves as a definition of the companies that are the best fit for your business.” Clay’s version: “every ICP account is in your TAM, but most TAM accounts are not in your ICP.”

On personas, HubSpot gives the operating split: “Think of your ICP as your pre-qualification filter and buyer personas as your personalization guide.” Forrester adds the complication that makes personas insufficient on their own: the same VP of HR can sit in several buying groups and play a different role in each. The five common roles are champion, influencer, decision-maker, user and ratifier.

An account can be a perfect ICP match and still sit outside the active target list on capacity or timing, which is why HubSpot’s ABM system tracks “target account” and “ICP tier” as two separate properties, Tier 1 being “a great fit” and Tier 3 “acceptable, but low priority.”

A fourth distinction is between ICP and IPP. Force Management draws the line “ideal customer profile does not necessarily equal ideal prospect profile.” A prestigious account can match the profile and still be a bad near-term prospect on stakeholder complexity or contract timing.

Jake Fuentes at Cascade gives the test for whether a candidate segment is real: “An ICP must be a single market segment: a group of people for whom the value of solving a problem is roughly the same, and who can be reached in roughly the same way.” Equal value and equal reachability, both necessary. OpenView arrives at the same test independently: a good segment “gets value in the same way and buys in the same way.”

Meka Asonye at First Round shows the difference between a vague segment and a usable one. Bad: “Series B technology companies.” Good: “companies who are experiencing X, who look like Y, who have previously tried these three solutions and need the product to do A, B, and C.”

Fit vs intent: why an ICP needs two scores

I’d install a fit-versus-intent model first. Clay defines fit as whether a company should buy from you, derived from stable attributes like size and stack. Intent is whether it’s looking right now, derived from recent signals like funding or hiring.

A high-fit, low-intent account is a nurture target. A high-intent, low-fit account is a distraction.

Score the two axes separately so you can treat them differently, then combine at routing. Fit is the eligibility gate; intent and timing set the tier. Bombora puts it from the data side: only a small portion of your ICP is in market at any time, and intent data exists “not to define your ICP, but to reveal which accounts are actively researching and should be prioritized now.”

The compelling event is why a static firmographic profile underperforms a signal-laden one. RevOps Masters reports its win/loss result: “The single most predictive factor in win/loss outcomes is whether the buyer had an identified event or deadline driving the purchase. Deals with a compelling event close at 3-4× the rate of those without one.”

Clearbit’s routing rule shows what combining the axes looks like in practice: “High-intent visitors should speak to sales right away if they have high fit. Low-intent visitors can be added to an ad retargeting campaign, email nurture sequence.” Their threshold example: an account that visited at least three relevant pages in a month, whose fit score qualified, got routed to the BDR team.

Which attributes belong in an ICP?

Four attribute families matter, and most bad ICPs stop at the first.

• Firmographic. The standard set: industry, company size, revenue range, employee count, geography and business model, plus ownership type. Bessemer’s bands: Commercial 1-500 employees, Mid-market 501-1,000, Enterprise 1,000+, with revenue bands of $0-1M, $1.1-10M and $10.1M+.

• Technographic. What they run: “which CRM they run, what marketing tools they rely on, where they host their infrastructure, and what sales platforms their reps use every day.” Two mid-market SaaS companies “might look identical on paper, but if one uses Salesforce with a modern tech stack and the other runs a legacy CRM with no API access, your approach should be completely different.” HG Insights goes further, measuring behind-the-firewall usage intensity and maturity, and separating Problem Aware, Solution Aware and Product Aware stages.

• Behavioral. Buying signals: content consumption, website engagement, event attendance, search intent. What separates companies that fit your ICP in theory from ones actively buying. In product-led motions this is where PQLs live, converting at 15-30% on signals like usage frequency, adoption of high-value features, usage growth week over week, and multi-product usage.

• Psychographic. How the organization decides. One list worth stealing covers “appetite for innovation, willingness to buy best-of-breed, attitude to vendors, decision-making style.” The pace difference matters too: “Some decisions require consensus across a large leadership team. Others sit with a single executive who acts quickly.” Lenny Rachitsky lists a company’s “way of working”, design-driven or operationally heavy, as an explicit field.

Prefer operational signals to generic firmographics wherever you can get them. First Round’s example is the cleanest illustration: instead of targeting companies with more than 500 employees, target “companies with more than 20 open, remote roles on LinkedIn. That’s a sharper ICP.” Same for pain: “It’s not enough to know the pain the prospect has, you want to deeply understand the impact of that pain.”

Bessemer’s five-factor needs framework, credited to Allyson Letteri, is the best short prompt list I’ve found for the situational half:

Pains: what pressing problems can your solution solve? Gains: what goals or desired outcomes drive your customer? Shifts: what company changes would make them open to new solutions? Blockers: what objections or misconceptions might hinder adoption? Motivators: what would urge them to move forward, ROI or testimonials or something else?

Shifts is the underused row. Common ones are “compliance mandates, a crisis or breach, new executives joining the team, or even when the team outgrows old systems.”

Product usage deserves its own note if you have a free tier. OpenView reports that Figma defined activation as “a user collaborates with other users in their first week” and modeled around 10 data points where “when two or more of these were triggered, there was a high likelihood for the account to upgrade.” A concrete multi-user trigger from the same playbook: when three or more users at the same company reach a high usage threshold, email the manager.

The jobs-to-be-done lens turns a tidy attribute list into a usable one. Tony Ulwick separates the job from the outcome, “a metric the customer uses to measure success”, which lets you cluster accounts by which desired outcomes are underserved and define segments that cut across industry and size bands. OpenView warns that conventional segmentation “can be severely deficient because it is an arbitrarily imposed view of the market,” and every criterion needs “a clear rationale as to why the particular criterion will impact or differentiate the needs or buying behavior of targets.”

Looker’s 2013-2015 ICP led with the job: “technical data teams, not end users or business analysts, who were starting to adopt cloud with large data sizes and complex analytical requirements AND the need to support a larger base of less technical end users.” Company size, 50-400 employees, appears only as a supporting filter and never as the explanation for demand.

How many attributes, and how narrow

Rachitsky collected the initial ICPs of more than a dozen B2B startups and found that “everyone landed on at least three attributes to describe their ICP. Some had more, but no one had fewer.” His instruction is to get “super-specific and super-narrow. Almost comically narrow.”

The receipts are the useful part:

• Gusto started with companies of five or fewer employees, in California, offering no benefits, salaried employees only, no other deductions, and willing to be paid eight days after running payroll.

• Gong picked three: selling in the US in English, selling via video conferencing, and selling software worth $1,000 to $100,000, “because beyond $100k, we assumed it was going to be a different sales cycle, and less than $1,000, it would be too transactional.” That left roughly 5,000 companies worldwide.

• Snyk went depth-first: “a developer building with Node.js who was very security-conscious.”

• Canva found theirs six months in, watching who got excited: social media managers and bloggers, “especially freelancers, who were building their own social media management business.”

At the other end, keep the filter to 4-7 attributes. More than 8-10 filtering dimensions and it is over-specified, fitting noise, and no longer usable by an SDR in real time.

Five frameworks for choosing a segment

Each framework contributes a different narrowing mechanism. You don’t need all five, but you should know which one you’re using.

In one line each: Moore picks the first segment by use-case urgency and reference community. Blank validates that a specific org type and buyer will actually purchase. Lean Analytics replaces aggregate counts with cohort metrics. MEDDPICC tests reality inside an account and aggregates into buyability patterns. Top-10 reverse engineering derives the profile from realized outcomes.

Moore. The transition that matters is early adopters to pragmatists who “want to see it proven out first, specifically in use cases that they themselves have and with customers they know and can reference.” A market is defined partly by reference behavior customers must “reference each other when making a buying decision.” Choose the first segment on problem severity rather than size,: “The size of the first pin is not the issue, but the economic value of the problem it fixes is. The more serious the problem, the faster the target niche will pull you out of the chasm.” The ICP has to encode the whole product, “the complete set of products and services needed to fulfill the compelling reason to buy,” which doubles as a test of whether you can actually serve the segment. Expansion then follows the bowling alley: target a connected segment that “by virtue of its other connections, creates an entry point into a larger segment.”

Blank. Customer Development runs discovery, validation, creation, company building. The earlyvangelist five criteria are the pre-ICP filter: “They have a problem. They understand they have a problem. They are actively searching for a solution and have a timetable for finding it. The problem is painful enough that they have cobbled together an interim solution. They have, or can quickly acquire, dollars to purchase the product.” He also advises going after companies that “aren’t the market leaders in their industries, but are fighting hard to get there,” then finding the internal evangelist who wants the competitive advantage. The validated sales model has to answer who influences, who recommends, who decides, who pays, where the budget sits, what acquisition costs, and how long a sale takes.

Lean Analytics. Cohort discipline instead of aggregate counts since “you can compare cohorts against one another to see if, on the whole, key metrics are getting better over time.” The OMTM is stage-dependent and temporary, and in interviews you “look for a subset of scores that spike; that’s your early-adopter customer segment.” Three cohorts, three questions: acquisition and win/loss (”can we get them?”), retention and GRR (”do they stay?”), expansion and NRR (”do they grow?”). The method needs an overfit warning: “if accounts that match your candidate ICP also churned at the highest rate, you have over-fit on a non-causal pattern and need to add an exclusion criterion.”

MEDDIC/MEDDPICC. Metrics, Economic buyer, Decision criteria, Decision process, Paper process, Identify pain, Champion, Competition, originally developed in 1996 by Dick Dunkel at PTC, with Paper Process and Competition added later. MEDDPICC contributes the feedback loop. Aggregate outcomes across won, lost, stalled and no-decision deals, tagged by account attributes, and you learn which characteristics predict buyability, cycle time and commercial success rather than theoretical fit.

Top-10 reverse engineering. Derive the profile from realized outcomes. SaaStr’s questions for your best deals: “How easy is it to close these top deals? How simple was it to set your customer up with onboarding and implementation? How sticky is your product?” A working version analyzes “10 to 20 best existing customers across revenue contribution, retention rate, sales cycle length, and product fit.” Allston Labs is the most rigorous: capture three layers per customer, “firmographic (who they are), behavioral (how they bought), and deal-shape (the buying committee that signed)... including the nulls, the distribution of nulls is itself a signal.” Weight the expansion cohort 3-5× because repeat purchase reveals organizational fit as well as product fit, treat closed-lost as “the empirical control group,” and note their headline claim: “the single highest-confidence ICP indicator is per-customer ACV growth at 12 months.”

How do you build an ICP from data?

Six analyses, in the order to run them.

1. Closed-won and closed-lost

Build a joinable dataset across six layers: opportunity, account, contact, source and campaign, product and use case, and post-sale retention. Separate new-logo, expansion and renewal cohorts before analyzing. Use a two-level loss taxonomy: primary as competitive, no decision, price and budget, or product gap; secondary as feature depth, integration ecosystem, implementation timeline.

Pull the last 12-24 months. No-decision outcomes are 25-40% of closed-lost deals on their own and should never be merged into competitor or price losses.

How much data you need before believing the pattern: 5-7 closed-won deals support a provisional hypothesis and nothing more, 30 closed-won is a stronger first-pass minimum, and 50 combined closed deals supports general win/loss pattern analysis.

Two behavioral signals worth pulling in the same query: hiring activity is associated with a 2.4× conversion lift, and funding events indicate a 3-6 month timing window. And triangulate rather than trusting one source: RevOps Masters recommends CRM loss reasons, rep debrief surveys and buyer interviews together, because rep-entered reasons measure perception.

A defensible scoring starting point, if you want one before you have enough data for a model: a 0-100 scorecard weighted firmographic fit 30, technographic fit 25, behavioral signals 25, deal potential 20.

2. Win rate and cycle length by segment

Never report blended. The example that makes it obvious: “A 24% aggregate win rate breaking down to 41% for SMB inbound and 9% for enterprise outbound tells you everything.”

The formulas, so the segmentation happens before the math: win rate by count is won deals divided by won plus comparable closed-lost, and sales velocity is opportunities × deal value × win rate ÷ cycle length. Segment by market size first, then calculate.

Across 4.2M opportunities at 530 companies and $54B of revenue, Ebsta and Pavilion found top performers had 42% shorter cycles, 76% higher ACV and 43% higher win rates than average. Early decision-maker involvement in the first two stages lifted win rates 55%, high-intent accounts closed at 3.4× velocity, and top performers were 24% more likely to disqualify non-ICP deals early. For orientation, average B2B win rates run 15-25% with mid-market cycles of 30-90 days, and lost deals take 2.0× longer than won ones.

One data-quality caveat before you trust any of your own numbers: Clari Labs, across 10M opportunities at 121 global enterprises, found 98% of companies fail to track closed-lost reasons consistently.

3. Unit economics by segment

Read gross-margin-adjusted CAC payback together with net dollar retention rather than alone, since a segment with short payback but poor retention is misleading. The formula: sales and marketing spend to acquire new customers ÷ (contracted new ARR × gross margin) × 12.

Bessemer grades it good at 12-18 months, better at 6-12, best at 0-6, with segment targets in favorable conditions of SMB under 12, mid-market under 18, enterprise under 24.

OpenView measured it:

by target customer, as good (50th percentile) against great (80th): under 20 employees, 9 months against 2; SMB at 20-100 employees, 7 against 4; midmarket at 101-1,000, 14 against 7; enterprise above 1,000, 14 against 9.

KeyBanc’s 2024 survey puts fully-loaded payback at 25 months in 2022, 21 in 2023 and 20 estimated for 2024. These figures disagree because the years, samples, ARR scales and definitions differ. Shape, not truth.

4. Retention cohorts

GRR is the stickiness check, NRR the expansion check, and a strong NRR can hide widespread churn behind a few large expansions. The formulas: NRR includes expansion and reactivation and can exceed 100%; GRR excludes expansion and caps at 100%.

Benchmarks worth arguing with: Bessemer’s good/better/best is NRR 100/110/120%+ with logo retention above 85/90/95%; best-in-class B2B NRR sits at 110-125% with median monthly churn of 3.7% for $1-3M ARR companies; ICONIQ’s 2025 report has software NDR settling around 110-120%; and OpenView’s 2023 data shows PLG companies at 105% NDR against 98% for non-PLG.

The number that justifies the whole exercise, from OpenView: “annual retention rates can vary from 50% to 90% across different customer types for the same product.”

In Clearbit’s worked example, ~86% of long-term revenue came from ~18% of leads entering the funnel, and they narrowed the ICP accordingly.

5. Usage and time-to-value

Median SaaS activation is 30%, average 36%, and activated users should retain at 2× non-activated. For B2B, activation typically takes two to three weeks rather than minutes. Similarweb scores PQLs on two factors, ICP attributes (company size, role, use intent) and usage behavior (frequency and breadth), combining them to set engagement level and which product to sell.

Strong-fit usage looks like fast time to value, high activation against matched non-ICP cohorts, repeated use of the core workflow, adoption of the differentiating feature, multiple active users or departments, and usage rising over time.

Weak fit looks like signup without ever reaching the aha, shallow single-feature or one-off use, low account breadth, frequency declining after the trial, a high support and onboarding burden, and no expansion path.

6. Interviews

Quantitative data alone misses the mechanism, and there is a number for how badly. Clozd reports that buyer and seller reasons for closed-lost deals “only align 15% of the time, meaning 85% of CRM closed-lost data is potentially inaccurate,” and that roughly 70% of buyers name a different primary competitor than the CRM has. Surveys alone don’t fix it either: response rates in some B2B sectors run under 2%.

The thresholds worth knowing: five to eight conversations for a focused discovery sprint, ten similar deals as the minimum cohort to see patterns, and twenty within a segment before making a major strategic change.

Jason Sippey, formerly VP Product at Twitter, describes the progression: “After the first 10, you start to see patterns. After 20, you really understand segmentation of the market. After 30, you have a really good understanding of what you actually need to go build.” His four questions: do you have this problem, how are you solving it today, how much are you spending to solve it, how does it impact your business.

For loss interviews specifically, GTM Operations’ five questions are better than most scripts: what triggered the search, which vendors made the shortlist and what were the first impressions, rank the top three factors in the decision, was there a moment your opinion of us shifted, and what would have had to be different for the outcome to change.

Two discipline rules. Buyers should speak 90% of the time. And the payoff for keeping at it: companies running win-loss for two or more years report an 84% increase in win rate.

The same interviews give four qualitative signs that you are closing in on the right segment: a significant jump in conversion rate, a significant jump in enthusiasm, a much stronger desire to act now, and the nod.

Sequence: quantitative analysis to find anomalous or attractive cohorts, qualitative interviews to find the mechanism and the language, then a second quantitative test for prevalence and predictive power.

Working with a small sample

You will almost always be here first. The workflow with roughly 12 accounts: pull the top 3 by revenue and top 3 by NPS or NRR, find the shared attributes, and treat the result as a hypothesis you revise quarterly.

Then borrow four rules from Allston Labs: use closed-lost as the control group, since attributes that differ between won and lost carry the decision weight; compute a win/loss ratio per attribute, where something appearing in 70% of won and 30% of lost is high signal; weight expansion cohorts 3-5×; and run three to five structured interviews with your most engaged customers.

What should an ICP document include?

Assembled from James Doman-Pipe’s GTM Playbook template, Full Funnel’s ABM six pillars and HubSpot’s partner ICP worksheet.

1. Header. Name, owner, last updated, change log. HubSpot’s is the only one of the three with a change log, and skipping it is how a definition goes stale without anyone noticing.

2. Firmographics. Industry, employee count, revenue band, geography, business model, funding stage and growth rate.

3. Technographics. Current stack, competitive and complementary solutions in use, IT spend and digital maturity.

4. Situational fit. Pain signal, buying trigger, current solution, budget parameters. Doman-Pipe’s bar for a pain signal is observable rather than adjectival,: “Not ‘they have messy data’ but ‘their team is manually reconciling data from three tools every Monday morning and it is taking four hours.’ Specific and observable.”

5. Buying committee. Economic buyer and what they care about, champion and the pain they feel, influencers, blockers and insiders, and day-to-day users versus buyers.

6. Why we win here. Two or three outcomes delivered, plus the structural advantage in this segment. If you can’t name the advantage, this is a segment you can serve rather than one to commit to.

7. Disqualifiers. Their own section, not the inverse of the positive criteria.

8. CRM fields. Fit score 0-100 and ICP tier as properties on the company object, how we find them, owner, last review date.

Published templates worth copying

• a16z’s nine fields: company size or revenue range, type of business, geography, industries served, job titles, a definable solvable problem, company-specific attributes, specific technologies used, unique buyer behaviors. With five diagnostic questions: which customers get the most out of the product, what traits do the best share, what objections recur in losses, who is easiest to upsell, and what do competitors’ customers have in common. Their worked example: “Large global retailers with onsite developer teams that use GitHub; have specific requirements for PII and customer data residence drive vendor decisions; and their cloud usage means they don’t have on premise requirements.”

• Lenny’s nine fields: company size, job title, pain point being solved, company’s unique way of working, specific tech used, type of business, price point, geography, and a unique place the user spends time.

• Bessemer’s Six Ps worksheet: Persona, Problem, Proposition, Product, Positioning, Promotion, with a bar for the persona section I like a lot: if all the criteria are met, the account should buy 80% of the time. Their worked example rejected universities as an ICP because “the higher education market is incredibly fragmented, insular, has restrictive budgets, and quite political,” and moved to residency teaching hospitals, clinics and paramedic schools.

• Productboard’s seven dimensions: company size, product stage, digital-first or digital transformation, single or multiple products, size of the product organization, care for the customer, B2B or B2C. Which resolved to “an early stage, digital-first startup with strong customer-centric, product-led culture, with one product team, building a single (ideally) B2B product.”

• HubSpot’s own 2024 NAM/EMEA ICP, which is rare in publishing its own numbers: 150+ employees, $100M-$500M revenue, software, IT and financial services, $4.6K average monthly deal size, 25% conversion rate, 75-90 day deal length.

• Adam Schoenfeld’s paragraph format, for teams that won’t read a table: “We are best for B2B SaaS companies with 75-5,000 employees in the US and Canada. Our ideal customers are growing revenue with a sales-led motion, have 10+ AEs assigned to account-based territories, have sophisticated revenue operations...”

• GTM Labs’ worked example is the one I’d hold up as the standard for specificity: 100-500 engineer companies on Kubernetes spending $5K+/month on Datadog, where a Senior SRE is the technical champion and the VP Engineering owns the budget, triggered by a Datadog renewal within 60 days.

The disqualifier section

A negative ICP is a distinct, named section. João Coelho treats them as separate architecture: “Disqualification criteria are not the inverse of ICP. They’re a separate architectural decision.”

Three exclusion types: hard (never target, such as a size floor or a country you cannot legally support), soft (target only with a different motion, such as enterprise logos needing field sales rather than cold email), and conditional (target only if a trigger overrides the default, such as a smaller company that just raised).

Concrete dimensions people actually write down: pre-revenue companies with no paid motion, or sub-200 employees when the deal size is too small; agencies reselling without their own list, or project-based engagements under $25K; a champion with no budget authority and no path to a decision-maker; missing core integrations; and a company eight months into a three-year contract with a direct competitor.

Then make it mechanical. A multi-signal routing rule: three or more negative signals auto-disqualifies, two routes to manual review, one proceeds but flags in the CRM. And the enforcement point that matters most: “Do not rely on rep discretion. Use CRM views, enrichment rules, and sequence entry criteria. If a segment is truly a negative ICP, the system should block it before it reaches a sender account.”

The CRO at Consensus built the positive version of this out of win-loss data and called it a UCP, an un-ideal customer profile: “We now know exactly what a bad prospect looks like,” which they tied to a more streamlined sales process and lower churn.

Who owns it, and where it lives

Two artifacts, not one. The narrative version lives in a shared doc: Full Funnel publishes a Google Sheet, HubSpot a PDF worksheet. The enforced version lives in the CRM as an ICP fit score (number, 0-100) and an ICP tier (dropdown: Tier 1, 2, 3, Not a fit) on the company object.

Ownership goes to RevOps where that function exists, “or if you don’t have RevOps yet, whoever manages the CRM and the go-to-market cadence.” Earlier than that it belongs to the founders because “when you’re super early, everything should still be sales-led. You’re still figuring out your ICP and personas, and that requires talking to customers.” When conversion is inconsistent rather than absent, product marketing is the better owner. HubSpot’s cadence: “review it every quarter using fresh CRM data, closed-won deal patterns, and updated customer feedback.”

How do you score and tier accounts?

Four scoring models require progressively more data.

• Weighted point-based, the simplest thing that works. 500+ employees +50, Director title +10, industry, country and technology 1-5 each, negative values for non-ICP attributes. Clearbit describes the progression from basic firmographic filtering to weighted points to predictive models, which is the order to walk it.

• Weighted rubric. A two-layer model: industry 20, employee count 20, revenue band 15, tech-stack adjacency 15, geography 10, funding and growth 10, hiring 10, each scored 100/60/30/0 by band. Combined as Account Score =(0.4 × Fit) + (0.6 × Intent), with intent recency multipliers of 1.0× for 0-30 days, 0.5× for 30-90, 0.1× beyond. A published variant weights firmographics 40%, technographics 25%, intent 20%, behavior 15%, with 70+ qualified, 50-69 worth working, under 50 disqualified. Another is industry 25%, size 20%, tech stack 15%, growth signals 15%, org maturity 10%, geography 10%, negative filters 5%.

• Decision-tree look-alike. MadKudu’s model uses firmographic, demographic and technographic data only for fit, with behavior in a separate Likelihood to Buy model recomputed several times daily and combined as Lead Grade = 2× Fit + 1× LTB. Bands: Very Good 85-100, Good 70-84, Medium 50-69, Low 0-49. Validation targets: recall above 70%, precision as a 10× conversion ratio between top and bottom scores, rejection rate under 5% for top scores.

• Customer-specific predictive. 6sense trains a model per customer on firmographics, technographics and custom fields, and the model itself identifies which characteristics matter. Bands Strong 81-100, Moderate 50-80, Weak 0-49, updated daily.

Two rules apply across all four models. Hard disqualifiers stay binary: never let a high firmographic score buy its way past an exclusion. And if you have two or three segments with meaningfully different win criteria, build separate models, because blending obscures the signal.

Tiering

ICP fit is the eligibility gate; intent, timing and signal strength determine tier placement. Composite formulas exist if you want one: (ICP Fit × 0.30) + (Intent × 0.25) + (Engagement × 0.25) + (Revenue Potential × 0.20), or fit 30%, revenue potential 25%, intent 20%, engagement 15%, strategic value 10%.

What matters more is tying each tier to a motion and a volume rather than to enthusiasm:

Tier 1 is a perfect fit across all dimensions, $100K+ ACV in the worked example, 10-25 accounts per rep, custom outbound with exec engagement and ABM ads, five to eight touches a week, fully personalized.

Tier 2 is a strong fit with one or two minor mismatches, $25K-$100K, 25-50 accounts per rep, programmatic ABM with industry-segmented messaging, two or three touches a week, semi-personalized.

Tier 3 is a partial fit that clears the minimum, under $25K, 100+ accounts per rep, inbound and paid social and self-serve, one automated touch, templated.

For calibration, HubSpot runs Tier 1 at 20-50 accounts with “deep research and one-to-one customized outreach” and Tier 2 at roughly 200 with industry and persona personalization. ACV maps to load: 20-50 accounts per rep in field sales at $50K-$500K ACV, 50-150 in a two-stage motion at $10K-$50K. Mutiny shows what real separation looks like: Tier A converted lead-to-opportunity at 30%, Tier B at 8%, and Tiers C and D were “effectively rounding errors.” Oliver Jay started with 100 enterprise accounts at Asana and concluded “that’s too many. It should have been just 50.”

Tiers should move. Mosaic narrowed 100,000 accounts to 8,000 managed by nine BDRs, with accounts crossing tiers annually as funding, size, geography and stack change. One team re-tiered its entire customer base in three days on four signals (growth potential, AI maturity, engagement, account health), landing on 8 Tier A accounts and 21 Tier B.

Trigger events

Six categories, across 6sense, Bombora, ZoomInfo, Demandbase, Common Room and UserGems: capital and funding; leadership changes, where UserGems reports past champions “convert about 3x higher than normal leads”; hiring and job postings, which reveal strategic priorities; technology adoption or replacement, where HG Insights’ technographic flags produce up to a 5x conversion rate to opportunity; expansion news; and layoffs or contraction, which can indicate consolidation needs.

First-party intent has its own ladder, from Clearbit: high intent is pricing, SKU, trial and demo pages; medium is customer stories, solutions, partners and integrations; low is the homepage, about page and blog.

Measuring the ICP itself

Not just the accounts. ICP pipeline contribution as a share of opportunities by count and by dollars, win rate and deal velocity by tier, and renewal rate by tier.

Bessemer stages the reporting: “At $1M ARR it is common to look at total pipeline, but at $25-50M this pipeline should be segmented by ICP targets” and split new-logo from expansion. Coverage benchmarks to hold it against: 3-5× the ARR goal unweighted, 2×+ weighted in-period, and $8-10 of pipeline per $1 of marketing spend.

How should an ICP change by stage, and how often should you revisit it?

Pre-PMF: uncomfortably narrow. Paul Graham’s version is to satisfy “all the needs of a subset of potential users” rather than a subset of the needs of all of them. YC: “recruiting 10 customers who have a burning problem is much better than 1000 customers who have a passing annoyance.” Bessemer says to make it narrow “uncomfortably narrow, so narrow it almost feels too small,” to focus on one ICP at a time even when the product has broad applications, and to avoid targeting enterprises first. First Round’s Emery Rosansky sets the epistemics: “in the early days, you’re flying nearly blind with a small amount of data, so this initial ICP is not much more than an educated hypothesis.”

The obstacle at this stage is usually not ignorance. Clay’s Kareem Amin names it: “Often it’s not that you don’t know who the customer is, it’s that you’re not picking, you haven’t committed to one hypothesis over the other.” When Clay did narrow, it enforced the choice by cutting irrelevant features, removing non-tailored marketing language, updating internal docs and re-educating the team. Cadence here is event-driven, not calendar-driven.

Two findings from Rachitsky’s founder interviews pull in opposite directions. Most founders got their first ICP wrong, so the exercise is iteration rather than insight: Persona’s Rick Song ran it “17 times in the early days,” and Databricks had none at all, working with “a hospital that was using this stuff... folks that were using us to determine earthquake magnitudes using Twitter.”

Skipping the exercise can slow product-market fit. Mathilde Collin at Front: “We did not think about ICP. I wish we did earlier on. It’s one of my biggest mistakes.” Barry McCardel at Hex was forced into it by launch: “once we did our public launch, we started getting a flood of different people coming in. Filtering through the leads spurred me into putting a much tighter definition around qualification for deals.”

Series A/B: evidence-driven narrowing. a16z frames the stage “rapidly experimenting to refine your ideal customer profile and build repeatable sales motions.” Bessemer’s ARR ladder is the most specific guidance available: at $1-10M ARR, “aim to fulfill the product vision for your initial ideal customer profile,” sequencing product-market fit, then strong net retention in the ICP base, then broadening; at $10-25M, “it becomes important to start segmenting leads to the salespeople and sales teams who are best equipped to handle them”; at $25-50M, “crystallizing your ideal customer profile” becomes one of the most important tasks, and you finally have enough deals to find the pattern.

OpenView’s analytical method for this stage: “Pull all of your opportunities from the past year and clean the data. Look for the expected LTV per opportunity versus the CAC across different characteristics, such as industry, company size, buyer persona and use case.” With the caution that firmographics alone “does not drive any particularly important insights.”

Jason Lemkin puts numbers on when to split the team: begin segmenting “as early as 3 reps,” formalize three or four categories “after maybe $8m-$10m ARR.” Stripe added one segmentation dimension per year starting with size, and Jeanne DeWitt Grosser describes the failure mode: “You know you have a segmentation problem when the same account executive is talking to a million-dollar company in the morning and a billion-dollar company in the afternoon. Those sales processes have nothing to do with one another.” Cadence: a quarterly playbook review, and Mandy Cole’s instruction is to check it against reality: “Go through and see if it’s what you’re doing in the field, and if it’s not, update it to see the impact on pipeline.”

Enterprise: a new ICP, not an extension. Joe Morrissey at a16z warns against treating interest as fit: “the single biggest mistake I’ve seen companies make when moving upmarket is mistaking initial enterprise interest for product-market fit. Just because you have product-market fit for startups, SMBs, or mid-market customers does not mean you’ll automatically find product-market fit in the enterprise space.” Segment’s enterprise ICP was “larger enterprises that were B2C, multi-product, multi-brand, multi-subsidiary, and most importantly, data-literate,” narrowed further to those with “forward-thinking data champion buyers who were either decision-makers or could partner with a CMO to shape the decision criteria,” and it required outbound pipeline generation with 9-12 month cycles.

Budget for the new segment differently. Lemkin: “If you force your CAC in a new segment to hit the same ROI as your overall, blended CAC goals, you’ll never leave your core safe ICP,” so 80% of new customers should hit a sustainable CAC while the 20% in new segments “just try to barely break even,” with 24 months to converge.

At this size the data team becomes the reality check. Bessemer: “The data team will come in and identify the users who have high product engagement, retention, and reach, and in many cases, the attributes of this cohort of users will differ, at least somewhat, from an original conception of the ideal customer.” And a16z’s point about who holds the pieces: “You can’t define your ICP in a silo. RevOps has data. Product sees patterns. Sales hears objections. Marketing picks up signal. Success sees who churns and who expands.”

Review versus revision. The cadence argument resolves once you separate the two words. Review is checking evidence: monthly or quarterly against live closed-won data, which is what Anis Bennaceur at Attention.com means by “the teams compounding fastest are refreshing it off live closed-won data on a monthly or quarterly loop.” Revision is changing the operating definition, and it should be rare and trigger-based: a new product capability that changes who you can serve, retention, win rates, service costs or willingness to pay differing by segment, a move upmarket, or behavioral data diverging from the original conception.

The apparent disagreement in the sources resolves the same way. Daria Dovzhikova supports full rebuilds after major launches or funding rounds; Erik Miller warns “Don’t change the ICP every quarter; that creates whiplash for sales and marketing.” Both are right, about different words.

How the ICP shows up downstream: positioning, outbound, roadmap, board

A definition nothing enforces isn’t a definition.

Positioning. April Dunford defines positioning “how your product is a leader at delivering something that a well-defined set of customers cares a lot about,” and the order of her five components is the argument for where the ICP sits: competitive alternatives (”what would customers do if our solution didn’t exist?”), differentiated capabilities, value, then target customer segmentation, then market category. The ICP is derived from unique value rather than assumed before it, because “your best-fit target customers are customers that really care a lot about your unique value.” Starting from market category is “the tail wagging the dog.”

The payoff, in her words: “Customers that are well suited for your offering will easily understand your value, will purchase faster, and are much less likely to ask for discounts.” The canonical case is repositioning a general enterprise CRM, which invited comparison with Siebel, as a CRM for investment banks where modeling interpersonal relationships was mission-critical: “from under $2M to close to $80M” in 18 months.

Demand generation. Chris Walker’s paid-social targeting formula is just “(1) Job Title + (2) Company firmographics (named accounts, company size, industry),” with the classic errors being no job-title targeting, no firmographic layer, and LinkedIn audience expansion that makes you “pay high CPMs to people you don’t want to reach.” Refine Labs allocates 60-70% of paid media to demand creation and about 30% to capture.

Budget for the measurement gap while you’re at it. Across 620 declared-intent conversions and $21.5M in closed-won ARR, Refine Labs found a 90% gap between software attribution and first-party self-reported data: podcast was credited with 53% of revenue self-reported and 0% by software.

Three worked outcomes. Clari moved to target account lists, persona alignment and zero-click content instead of lead-gen forms: ad spend down 38%, cost per SQO down 36%, acquisition cost down 67%, win rates up 64%. Loxo built priority account lists with buyer-count exclusions and ICP-focused content: acquisition cost down 23%, ARR up 45% quarter over quarter. Zappi analyzed region, industry and persona out of the CRM and targeted by job function: 3× average deal size and 7× qualified pipeline against spend.

Outbound. Clay’s method is the ICP definition and the list build in one motion: “Pull the deals that closed fast, stayed, and expanded, and ask what they had in common the quarter before they bought. The patterns that repeat are your ICP. Write the definition down as a list of filters you can source against later, split into firmographic (size, industry, geography), technographic (the tools they run), and signal-based (funding, hiring, expansion).”

Apply disqualifiers before list export since “disqualifiers applied after export waste enrichment credits and create compliance risk.” The workflow most teams end up with: build criteria in Sales Navigator, export, run through Apollo or ZoomInfo for contact data, layer Crunchbase for funding, overlay BuiltWith for technographics, process through Clay for dedup and scoring. With one caveat worth pinning up: “Company headcount is the least reliable filter in Sales Navigator. Many companies mischaracterize their size or list no headcount at all.”

Content. The operating chain is ICP and account fit, then priority personas and buying situations, then topic pillars, then stage-appropriate formats, then channels. Every campaign, product and content strategy should align to at least one persona, built on “motivations, goals, pain points, and language, not only demographics.” Exit Five’s practical step is the one teams skip: “Create a One-Pager: Summarize your ICP in one clear document that your entire team can use.” For volume calibration, 47% of buyers view three to five pieces of content before talking to a rep.

Roadmap. The ICP “should be a formal input into quarterly roadmap reviews, not just a Marketing document.” A segment-weighted score makes the tradeoff explicit: revenue × strategic fit × growth signal × usage depth. Worked: a $30K mid-market account that is a core ICP match, expanding, and a power user scores 108; a $100K enterprise outlier that is contracting and lightly used scores 5, making “the mid-market account’s feedback 21x more valuable for prioritization purposes.” The counterweight keeps it honest: over-indexing on power users “harms the product experience for the rest of the user base, and steals product effort away from features that could go towards acquiring new users.”

Board reporting. a16z flags Net New Weighted Pipeline as the leading indicator of market pull. Pair each metric with the decision it drove, since 70% of a board meeting should focus on the future and the issues at hand. First Round’s Vanta example is the model: a 6% win rate on one product-buyer combination where “generally, a win rate below 30% means salespeople are wasting time and you need an unsustainable amount of pipeline. In this case, the solution wasn’t to fix sales, but to immediately stop selling this product to this buyer.” And SaaStr’s observed NRR gradient by deal size makes the segment story concrete: single-seat deals under $99/month at roughly 3% monthly churn, $99-$999 at about 100% NRR, $10K-$100K+ at about 120%.

Why do ICPs fail?

Fullcast and Pavilion’s 2026 GTM benchmark found ICP misalignment cuts win rates by up to 75%, and that 63% of 118 CROs had little or no confidence in their own ICP. Nobody in that survey is missing a document. They just don’t believe theirs.

Nine failure modes, in the order I’d expect to hit them rather than the order they’re usually listed.

1. Written but not operationalized. The pattern: “ICP alignment is not a strategy doc... You defined your ICP in a slide deck three quarters ago. But you haven’t updated routing rules since.” Owner.com’s CRO Kyle Norton found “either bad or no targeting on prospects,” poor-fit closes and mis-set expectations, with churn a “massive drag on efficiency” in his first 90 days; his fix was to pause the Growth plan, say no to a lot of customers, and build a lead quality system. Remedy: audit the top 50 closed-won deals against current scoring and routing rules. Figma’s version was to segment sales at the 1,000-employee line and pipe product usage into Salesforce via reverse ETL.

2. Static. One RevOps lead describes a client that “re-wrote their ICP properly last year, negative criteria and all, then left the old scoring weights and routing rules untouched for two quarters. The doc said one thing, the system kept qualifying the drifted accounts, and nobody reconciled the two until win rate forced it,” calling drift a “silent revenue killer manifesting as declining win rates, lengthening sales cycles, and churn concentration.” Lemkin’s line: “an annual ICP exercise is a 2021 habit.” Remedy: treat it as a living hypothesis with a monthly closed-won review. One founder who did that cut outbound from 2,000 to 400 emails a month while win rate rose from ~3% to 11%.

3. Aspirational. The written ICP describes the customer the company wants; closed-won describes the one it has. The cost: “The gap between those two is where marketing spend goes to die,” with an SDR team hitting an 11% win rate and a 178-day cycle against the written ICP. Sometimes the gap is a level, not a segment: First Round documents a company that “thought they were selling to CISOs at early- and growth-stage orgs, but realized through user research discussions that the people that acutely felt the pain were 2-3 levels below a CISO in an Enterprise org.” Remedy: start from 18-24 months of closed-won, and note Rachitsky’s finding that data from outbound sales beats leads from investors and friends as a signal.

4. Too broad. Cascade’s Jake Fuentes defined “nontechnical business analysts using Excel to crunch big data sets,” later calling it “much too broad” and describing the result as ICP fray: a scooter company managing location data, an HR team and a retailer under one profile. “To know what problem you’re solving, you need to know who you’re solving it for.” Remedy: replace generic firmographics with operational signals, and start from about three distinctive attributes.

5. Demographics instead of behavior and situation. A profile of static firmographics with nothing distinguishing accounts likely to buy now. Remedy: convert filters into real-time signals, and mine what you already own. Lemkin: prospect call transcripts are “the highest-signal first-party data your company will ever own.”

6. No negative ICP. No-decision deals “can account for up to 40% of a pipeline,” which is exactly what disqualifiers are for. Remedy: Mandy Cole’s Green/Yellow/Red: green is proactive pursuit, yellow is inbound and referral only, red is no-go.

7. No discipline at close. Quota pressure closes poor-fit deals. Katrina Wong separates an ICP from inbound demand: an ICP is a choice about “who you want to sell to versus who just wants to buy from you.” Clay’s co-founder Varun Anand gives the honest version of what discipline costs: “We had few true customers because almost none of the existing ones fit our new ICP,” and after narrowing to outbound sales, “nearly all of our original customers churned.”

8. Confusing ICP with persona. Remedy: keep the split clean: “ICP answers ‘what kind of company should we go after.’ Persona answers ‘who inside that company makes the call.’” Qualify the account first, then map the committee.

9. Trusting CRM loss reasons. 15% buyer-seller alignment, 70% competitor mismatch, 98% inconsistent tracking, all cited above. Remedy: 20 interviews within a segment before concluding anything.

The single best test of whether yours is real: an SDR can identify or disqualify an account quickly, marketing can target it, RevOps can score and route it, and sales can explain why it should be pursued now. If any of the four can’t, it’s a point of view rather than an ICP.

Where the practice is heading

The ICP-as-PDF is explicitly retired in current practice. iCustomer, July 2025: “The traditional definition of an Ideal Customer Profile as a static document describing your perfect customer is obsolete. In 2025, an ICP isn’t a document, it’s a dynamic, data-driven automated system that evolves in real-time based on live signals and market intelligence.”

Tooling now derives the profile directly. ZoomInfo’s AI-Generated ICP takes a CRM report of won and lost deals and learns which attributes indicate best targets, refreshing as conditions change. HG Insights’ Market Analyzer Copilot builds ICPs from win/loss, firmographic and technographic data correlated with highest-CLV customers.

The capability gap is the more interesting number. Enlyft found 81% of respondents likely to update their ICP in 2024, yet nearly 70% lacked a strong grasp of account scoring and 43% had invested in scoring without understanding its efficacy. The differentiator is not access to a model. It’s knowing what a good one looks like.

Further reading

• Lenny Rachitsky, How to identify your ideal customer profile. The operator counterpart to this page, and the best collection of initial ICPs anywhere: a nine-item attribute picker, a dozen founder accounts, and the three-attribute floor. It paywalls at the template and the comparison chart.

• a16z, a framework to define and refine your ICP, for the nine-field template and the revision triggers.

• First Round Review, the most common go-to-market questions from founders, which is where the hiring-signal example and the “educated hypothesis” framing come from.

• April Dunford, a quickstart guide to positioning, for why the ICP comes after differentiated value rather than before it.

• Full Funnel’s ICP for ABM, which publishes an actual Google Sheet.

• Allston Labs on closed-won deconstruction, the most rigorous small-sample method I found.

• Clozd on building a win-loss program, for the interview thresholds and why CRM loss reasons mislead.

• Clay’s account of what narrowing costs, which is the least comfortable line in the whole set: after they narrowed, “nearly all of our original customers churned.”

FAQ

What is an ideal customer profile? An account-level definition of which companies you are best at winning, serving, retaining and expanding, expressed as scored criteria and explicit disqualifiers rather than prose. It answers four questions: who will buy, who will buy smoothly, who can be implemented and serviced, and who will renew and expand.

How is an ICP different from a buyer persona? The ICP is the company, the persona is the person inside it. Qualify the account first, then map the buying committee. Treating them as synonyms produces a profile that can’t be used for routing or list building.

How is an ICP different from TAM? TAM is a dollar-value estimate of the whole opportunity. An ICP is a qualitative definition of which account types fit best. Every ICP account sits inside your TAM; most TAM accounts are not in your ICP.

How many customers do you need before you can define one? 5-7 closed-won deals support a provisional hypothesis, 30 is a stronger first pass, and 50 combined won and lost supports general pattern analysis. Below that, use the top 3 by revenue and top 3 by retention as a quarterly-revised hypothesis, with closed-lost as the control group.

How many attributes should an ICP have? Four to seven, and at least three. Past 8-10 dimensions you are over-specified and fitting noise, and the profile stops being usable by an SDR in real time.

How often should you update an ICP? Review monthly or quarterly against live closed-won data. Revise the operating definition only on a trigger: a new capability that changes who you can serve, retention or unit economics splitting by segment, a move upmarket, or behavioral data diverging from the original conception. Rewriting it every quarter creates whiplash in sales and marketing.

Who owns the ICP? One named owner with a change log, usually RevOps or whoever manages the CRM and the go-to-market cadence, with input from product, sales, marketing and customer success. At the earliest stage it belongs to the founders, because it is still being discovered in customer conversations.

Do you need a negative ICP? Yes, as its own section rather than the inverse of the positive criteria. Split hard exclusions you never target from soft ones you serve on inbound and referral only, and conditional ones a trigger can override. No-decision deals can reach 40% of a pipeline, and disqualifiers are what protect it.

Where should the ICP document live? Two places. A shared doc for the narrative version, and the CRM for the enforced version, as a fit score and a tier on the company object. If it only exists in the doc, it isn’t operating.

What is the difference between fit and intent? Fit is whether a company should buy from you, from stable attributes. Intent is whether it is looking right now, from recent signals. Score them separately and combine at routing: fit sets eligibility, intent and timing set the tier.

How do you know the ICP is working? Track ICP share of pipeline by count and dollars, win rate by tier, deal velocity by tier, and renewal rate by tier. If Tier 1 doesn’t convert materially better than Tier 3, the tiers are decorative.

When we incorporated GrowthX in 2025, we deliberately ran the work by hand first as a content marketing agency for clients like Ramp, Lovable, Vercel, Brex and Reddit — before building any external facing product. That got us to 8 figure revenue and nearly 100 people team.

In June 2026 we launched the first user-facing version of our product. Launching a new product inside a company this size should be like getting back to seed stage, but with a different set of pros and cons .

A larger engineering team lets us take bold steps. It also means I have to keep reminding myself that the team building this first version should behave like it’s seed stage – that means being conscious on how to handle Conway’s law, data migrations, rollouts, etc.

I’ve kept rereading or referencing Paul Graham’s essays. And I wanted an audio version for my commute.

So, I pointed the same pipeline I built for a Computer History Museum oral history at Paul Graham’s blog to get the list I want to read again. This is the batch, grouped by theme.

Sharing here in case others find it useful.

09-07-2026 Update: I’m switching all the essays to use Eleven Labs + did some post processing to make the listening easier, and added chapters + enabled download so you can listen to the large “whole theme” volumes with any audiobook app of your choice.

Note: The audio is machine-narrated, and the text is adapted for listening rather than read verbatim. Every episode links to the original, which is the real thing.

Download the omnibus audio book format

Here’s the full 25 hours audio book version with chapters that you can load into any audio player of your choice:

👉 https://drive.google.com/file/d/1FyxUePBx7Koajk_hlrHO-hlVyYxyUWaT/view?usp=sharing

Startups & Founders

Whole theme: 14 hours.

How to Get Startup Ideas

Startup ideas come from noticing a problem you have yourself, not from trying to invent one. Every good idea shares three traits: the founders wanted it, could build it, and few others saw its value yet. A social network for pet owners sounds plausible but has zero real users; Microsoft’s Basic for the Altair had only a couple thousand possible buyers who needed it badly.

Pick depth over breadth: a narrow group that wants something urgently beats a wide group that finds it mildly interesting. Facebook started at Harvard, a few thousand students, before spreading through every college and then everyone. Stripe succeeded by not flinching from the “schlep” of payments that scared other programmers off.

This is for anyone hunting a startup idea who keeps landing on ones that sound fine but fold under scrutiny. The advice: get to the leading edge of some fast-changing field, then build whatever seems to be missing from your own life there.
Duration: 40min, Publish Date: Nov 2012, Post: paulgraham.com/startupideas.html

Do Things that Don’t Scale

Startups don’t take off on their own; founders force the launch by hand, then automate once it’s rolling.

Stripe’s founders got early users by saying “give me your laptop” and installing the product on the spot, a technique YC now calls a Collison installation. Airbnb’s founders went door to door in New York shooting “professional” photos of hosts’ apartments, thirty days that separated success from failure. Wufoo sent hand-written thank-you notes to every new user, and Facebook stayed Harvard-only, then school-by-school, before opening up. Pebble hand-assembled its first watches before selling $10 million worth on Kickstarter, and Viaweb’s founders built stores for merchants who wouldn’t build their own.

It hands anyone starting something the same reframe: judge an early product by whether the founders are doing the right small things, not by whether it looks like it’s taking over the world yet.
Duration: 25min, Publish Date: Jul 2013, Post: paulgraham.com/ds.html

Startup = Growth

A startup is defined by one thing: growth, fast enough to matter. Being new, working on technology, or taking venture money are side effects, not the definition. A barbershop can satisfy plenty of demand but can’t reach more customers than its chairs allow; a search engine has no such ceiling.

Growth rates make the case concrete. During Y Combinator, 5-7% weekly growth counts as good, 10% as exceptional, 1% as a warning sign. A company making $1,000 a month growing 1% weekly reaches $7,900 a month in four years; at 5% weekly it reaches $25 million. Wozniak building his own computer in 1975 and Page and Brin fixing search they found inadequate both turned personal problems into markets nobody else had noticed yet, and both became defensible once the wider market caught up.

That same growth number explains why VCs fund unprofitable companies, why founders take money they don’t strictly need, and why acquirers like eBay buying PayPal pay a premium: a fast-growing company is valuable and, left alone, dangerous. Useful for anyone trying to tell whether they’re running a startup or just a new business.
Duration: 30min, Publish Date: Sep 2012, Post: paulgraham.com/growth.html

Schlep Blindness

Startup founders avoid great ideas not because the ideas are bad, but because the work behind them looks unpleasant. A tedious, unglamorous task gets shut out of consideration before it’s even weighed, and the mind does the shutting out without telling you.

Stripe makes the case. For over a decade every hacker who processed payments online knew how painful it was, yet they built recipe sites and event aggregators instead, because fixing payments meant striking deals with banks, handling fraud, and navigating regulations.

The fix: stop asking what problem to solve, and ask what problem you wish someone else would solve for you.
Duration: 5min, Publish Date: Jan 2012, Post: paulgraham.com/schlep.html

Alive or Default Dead?

Half the founders running startups past their first year don’t know whether their current trajectory ends in profitability or in running out of cash. Default alive or default dead is the question, and answering it decides everything else: ambitious new bets, or a rescue plan.

The trap is the fatal pinch: default dead, slow growth, not enough runway left to fix it, reached because founders counted on raising more money instead of checking the math. Steep growth, over 5x a year, buys real investor interest; anything less makes fundraising a gamble, not a plan. The bigger killer is overhiring: founders raise a round, growth is only so-so because the product is merely appealing, and they hire to force growth that a bigger team can’t produce. Airbnb went the other way, waiting four months after Y Combinator to make its first hire.

For anyone past nine months in, this gives a five-minute gut check before the next hiring decision, and a reason to write down plan B before plan A stops working.
Duration: 8min, Publish Date: Oct 2015, Post: paulgraham.com/aord.html

What Happened to Yahoo

Yahoo’s collapse traces to two problems Google never had: easy money and confusion about what kind of company it was.

By 1998 advertisers were overpaying for banner ads, and Internet startups were buying traffic with money raised on the strength of Yahoo’s own revenue growth, a loop that made search look worthless. David Filo dismissed search as 6% of traffic in an industry growing 10% a month. Yahoo also called itself a “media company,” renaming programmers’ work “producers” and “properties” to dodge Microsoft’s attention, and let product managers and designers run the show while programmers just implemented. Google, at the same 500-person size, still asked engineers “what should we do?”

The lesson: any company that needs good software needs a hacker-centric culture, not just ones that call themselves technology companies. Bad programmers compound, since good ones won’t work alongside them, and there’s no recorded recovery once that spiral starts.
Duration: 12min, Publish Date: Aug 2010, Post: paulgraham.com/yahoo.html

Ramen Profitable

Ramen profitable means a startup earns just enough to cover the founders’ living expenses, not enough to count as a real business. That distinction matters because it removes the need to raise money to survive.

A hardware company might spend $50 million over five years before turning a profit, and that profit signals real success. A startup profitable on $3000 a month after two months, run by two 25 year olds living cheaply, signals nothing about scale, but shares the same freedom from investors. That freedom pays off four ways: better terms since investors can’t stall you into desperation, more investor interest since it proves people will pay, morale since the future flips from default-dead to default-alive, and no lost months of productivity spent thinking about a raise instead of the product, the way Y Combinator itself found when raising its own money.

It does not mean bootstrapping forever, and it can tip into disguised consulting if the revenue never becomes a scalable product. Google started this way, licensing search to Yahoo before building its real business.
Duration: 11min, Publish Date: Jul 2009, Post: paulgraham.com/ramenprofitable.html

How to Start Google

Starting your own company is the way to get really rich, and you can start preparing for it at 15, before you have any idea or cofounders. Nearly everyone on the published lists of the richest people got there by starting a company, not by working for one.

Getting good at technology comes from working on your own projects, not from classes: Mark Zuckerberg built the Harvard facebook overnight because he was a programmer who noticed the university hadn’t gotten one online. Steve Wozniak just wanted his own computer; Larry and Sergey just wanted better search results than “every page with the word rugby.” Ideas come the same way, from noticing broken things once you’re good enough to fix them, and cofounders come from working alongside people, which is why university matters: not its prestige, but the difficulty of getting in.

The advice for someone 14 or 15 comes down to two things: build stuff, for yourself and your friends, and do well enough in school to get into a good university.
Duration: 15min, Publish Date: Mar 2024, Post: paulgraham.com/google.html

What Microsoft Is this the Altair Basic of?

A startup idea that sounds lame at launch is often the strongest signal of a big company forming. Practically every huge company looked wrong at first: a Basic interpreter for the Altair, strangers renting airbeds, a site for college students to stalk each other, a single-board computer that used a TV as a monitor, one more search engine among ten that were all de-emphasizing search.

Founders rarely know why their own idea is promising; instinct pulls them toward something the market is missing before they can explain it. The test is to ask “what Microsoft is this the Altair Basic of.” Sometimes the answer doesn’t come, sometimes there are multiple answers of wildly different sizes, as with a startup that could grow into three distinct Microsofts.

Since nobody can predict how big any of those paths gets, the advice is to chase whichever one feels most exciting right now, on the strength of the instinct that produced the idea in the first place.
Duration: 2min, Publish Date: Feb 2015, Post: paulgraham.com/altair.html

The 18 Mistakes That Kill Startups

Startups almost never die from one dramatic mistake. They die from not building something people want, and eighteen specific habits funnel founders into that outcome.

Single founders lack the friends who’d talk them out of bad calls and the esprit de corps that survives low points; Y Combinator sees this pattern in a fifth of founder splits. Bad platforms kill quietly: Hotmail stayed on FreeBSD rather than Windows, and PayPal’s near-switch to Windows would have cut performance to 1% of Unix. Raising VC money changes a company physically: it moves to an office, hires employees over founders, and locks in a direction, while investors like Google’s early board can still fire a Steve Jobs. Google and YouTube both won by building the product first and worrying about revenue later; YouTube’s founders got $1.6 billion for lessons Google could have bought for $10 million eighteen months earlier.

This is aimed at people about to start a company, not people running one: a checklist of ways the whole effort can fail before it gets the chance to succeed.
Duration: 29min, Publish Date: Oct 2006, Post: paulgraham.com/startupmistakes.html

Change Your Name

If your startup is named X and someone else owns x.com, the fix is to change the name, not to work around it.

Not having the matching domain signals weakness: a marginal domain reads as a marginal company, while owning it signals strength even when it has no bearing on the business, as with Stripe. Founders resist changing course for two reasons: attachment to the name as identity, and the belief there’s no better option, both false. Naming is a separate skill from founding one; in 20-minute office hours, a good name turned up 80% of the time. Almost any non-bad word or word pair works, and enough such domains sit cheap or unregistered to make a shortlist worth buying.

100% of the top 20 YC companies by valuation hold the .com of their name, 94% of the top 50, but only 66% of the current batch. That gap is a problem fixable in a couple of days, for anyone willing to admit the name isn’t sacred.
Duration: 4min, Publish Date: Aug 2015, Post: paulgraham.com/name.html

Ideas for Startups

Coming up with startup ideas feels hard mainly because people think an idea has to already be a million dollar idea. It doesn’t: most startups end up nothing like their initial idea, so the real value of that first idea is the process of finding out it’s broken.

Treat the idea as a question, not a blueprint, and it survives being wrong as long as it leads somewhere. Two ingredients feed the questions: exposure to new technology and the right friends, which is why universities and grad school produce so many founders, and why a coding job at a big company doesn’t. Spam filtering shows the other half: in 2002 everyone tolerated spam or trusted existing heuristic filters, and a simple statistical classifier solved it because nobody had actually tried.

Useful for anyone stuck waiting for a idea to strike, and for founders deciding whether to sit down and brainstorm or just build things with friends and see what breaks.
Duration: 21min, Publish Date: Oct 2005, Post: paulgraham.com/ideas.html

Don’t Talk to Corp Dev

Corp dev is the group inside a company whose job is to buy other companies. Talking to them makes sense in exactly two situations: the company is dying, so there’s nothing to lose, or it’s doing so well that any offer will have to be high and a slow walk gets a quick brush-off.

The trap catches companies in between, especially ones under a year old growing fast but not yet big. Corp dev opens with a lowball number to test resolve, then drags the process out for months, and even a signed price can get halved later when “the boss vetoes it.” A Google contact, asked what happened to “Don’t be Evil,” said corp dev never got the memo. The pull works because a flattering approach and a possible high number make a distraction feel like due diligence.

The fix is Rockefeller’s trick against his alcoholic grandfather’s fate: never take the first drink. Don’t take the first meeting unless the answer to “do we want to sell right now” is yes.
Duration: 7min, Publish Date: Jan 2015, Post: paulgraham.com/corpdev.html

Mean People Fail

Meanness and success rarely travel together. Among startup founders, programmers and professors, the most successful people are almost never mean; meanness looks common only because the internet finally lets everyone publish it.

Being mean makes you stupid: fights force you to think in tricks that work once, not ideas that generalize, so your brain spins without traction. Mean founders also can’t hire the best people, who have other options; and the ones who keep going past an acquisition offer are usually driven by wanting to improve the world, not by money. Historically success meant winning zero-sum fights over scarce resources, from nomads driving out hunter-gatherers to Gilded Age railroad men. Archimedes shows the older pattern: winning by ideas, killed anyway by a Roman soldier. What is changing is that new-idea games, once confined to mathematicians and writers, are spreading into the wider economy.

For anyone building something, an argument that meanness is a competitive disadvantage now, not just a moral one.
Duration: 5min, Publish Date: Nov 2014, Post: paulgraham.com/mean.html

Relentlessly Resourceful

A good startup founder comes down to two words: relentlessly resourceful. Relentless alone isn’t enough, because the obstacles in an interesting domain are novel; you don’t know in advance whether you’re plowing through foam or granite, so you have to keep trying new approaches rather than just pushing harder.

This isn’t the general recipe for success. Writing and painting call for active curiosity instead, since the obstacle there is your own obtuseness, not something external. After four years of trying to teach the quality to people, many, though not all, turned out to have a latent capacity for it, especially those who’d spent their whole lives under some authority’s thumb.

The test doubles as a filter: ask whether you’re relentlessly resourceful before starting a startup, and ask it of anyone you’d take on as a cofounder. It also bounds how many startups can exist at all, since the limit isn’t economic, it’s the size of the pool of people built this way.
Duration: 6min, Publish Date: Mar 2009, Post: paulgraham.com/relres.html

The Hardest Lessons for Startups to Learn

A startup should get a minimal version 1 out fast, then improve it constantly based on how users actually respond. Speed of iteration matters more than where the product currently stands.

Wufoo released its form-builder before the underlying database was ready; 83,000 people showed up anyway, and Linux users complaining about Flash usage led to a rewrite that wouldn’t have happened otherwise. Determination, not intelligence, is named the key trait in founders: Microsoft, Yahoo, and Google were all started by dropouts, not professors, and VCs who chase eminent academics are backing the wrong quality. Fear other unknown startups, not Google, and treat every “we want to invest” or “we want to acquire you” as something that won’t happen until it does.

This covers release strategy, competitive fear, deal-making psychology, and the case that a startup is best understood as a way to compress a lifetime of earning a living into a few intense years.
Duration: 26min, Publish Date: Apr 2006, Post: paulgraham.com/startuplessons.html

The Ronco Principle

Ron Conway has invested in more top Silicon Valley startups than any other angel or VC, and he did it by being good, not by seeming good. Referrals built the position: Google, Facebook, and a Twitter introduction that came straight from Evan Williams. Founders send him deals because he has never been caught treating anyone badly.

That pattern holds across investors generally: plot benevolence against returns and the line trends up, not down. Two forces make it hold. Startup dealings are transparent now, so mistreating a founder gets back to other founders fast, and Y Combinator’s cross-portfolio view makes investor behavior even harder to hide. And the world moves too fast to fake it selectively: the college kid you dismiss today might run the hottest startup in two years, so the only workable strategy is treating everyone well.

The reach goes past startups. Any world getting more connected and less predictable is heading toward the same rule: it becomes too costly to seem good without being good.
Duration: 4min, Publish Date: Jan 2015, Post: paulgraham.com/ronco.html

What We Look for in Founders

Determination matters more than intelligence in startup founders, as long as founders clear a basic bar of smarts. Founders hit constant obstacles and can’t get demoralized. But raw stubbornness needs flexibility too, the willingness to change the plan while staying determined to get downfield.

Bill Clerico of WePay kept calling bureaucratic partners until they gave in. Daniel Gross of Greplin dropped his original ecommerce idea, tried two more, and landed on Greplin days before Demo Day. Airbnb looked too crazy to fund until its founders described selling Obama and McCain branded cereal boxes to pay rent. Sam Altman’s suggested application question, asking about a time someone hacked a system to their advantage, became one of the most useful signals in reviewing founders.

Also cited: imagination that produces ideas at the right level of craziness, a naughty streak that ignores irrelevant rules, and a close, resilient friendship between co-founders, since startups pull apart any relationship that isn’t strong.
Duration: 4min, Publish Date: Oct 2010, Post: paulgraham.com/founders.html

A Student’s Guide to Startups

Starting a startup is becoming a real third option alongside a job or grad school, and it could become as common as grad school once was. Undergrads have the technical skill for it, but graduation itself changes something: once you leave school, a startup reads as your occupation rather than a summer project, so peer judgment pushes you to make it work.

The best age turns out to be the mid-twenties, not younger. Young founders gain stamina, poverty, rootlessness, and ignorance: Robert Morris and Paul Graham lived like 23 year olds while building Viaweb, charging $300 a month, an order of magnitude below the market. Sam Altman started Loopt after his sophomore year and became the exception, not the rule; Google, by contrast, produces zero startup founders because it’s too comfortable to leave.

This is for CS majors weighing a startup against a job or grad school, especially anyone hunting for a co-founder: school is named as the best place to find one, before classmates get tied down by mortgages and jobs.
Duration: 35min, Publish Date: Oct 2006, Post: paulgraham.com/mit.html

Be Good

A startup should aim to make something people want and not worry about revenue at first, which is basically the definition of a charity. Businesses that behave like nonprofits early on tend to become the most valuable companies.

Craigslist runs on a tiny staff and stays upwind of revenue it could easily take. Google ran ad-free for over a year, indistinguishable from a nonprofit indexing the web. Early Microsoft cut hardware prices by breaking IBM’s PC monopoly, and its stock rose like Google’s until it started treating users badly and went flat for years. Being good also works mechanically: it keeps founders working through the low points (Blogger’s Evan Williams alone in the office, still hosting thousands of blogs), it draws in investors, customers and the best hackers, and it gives 57 surviving Y Combinator startups a stateless rule for every hard decision: do whatever’s best for your users.

For anyone running a startup and drowning in decisions, this hands over a working compass rather than a slogan.
Duration: 15min, Publish Date: Apr 2008, Post: paulgraham.com/good.html

Before the Startup

Startups don’t work the way instinct predicts, and the mistakes founders make come from trusting instincts built for school and big companies instead. Y Combinator’s own function, joked about internally, was to tell founders things they’d ignore and then hear “I wish we’d listened” a year later.

Domain expertise beats startup expertise: Mark Zuckerberg succeeded as a “complete noob” at startups because he understood his users. Gaming the system, the trick that works in school and at big companies, stops working here; there’s no boss to impress, only users deciding if the product does what they want. Voice-over-IP, Airbnb, and Google all trace back to founders solving a real problem out of curiosity, not a business plan.

The advice for college students specifically: don’t start a startup at 20, since success then forecloses the “breadth-first” exploration years that people like Darwin (age 22, HMS Beagle) got to have. Learn things that matter, work with people you respect, and let the idea arrive as a side project.
Duration: 25min, Publish Date: Oct 2014, Post: paulgraham.com/before.html

Can You Buy a Silicon Valley? Maybe.

A city cannot copy Silicon Valley by funding startups the way Y Combinator does. Seed-stage startups are mobile: they take the money and move wherever the next round comes from. To build a local hub you need startups that put down roots, and that only happens with enough money that they never need to leave.

That price is at least half a million dollars per startup, twenty times a $15-20k YC check; Wufoo rooted itself in Tampa on $118k, but that’s an outlier. At a million each, a billion dollars buys a thousand startups and, within five years, a self-sustaining scene, enough that VCs start opening local offices instead of demanding relocation. The harder problem is picking which thousand: local VC funds look impressive to city officials but fail at exactly the skill that matters, so the proposed shortcut is offering relocation money to startups already backed by well-known Silicon Valley angels.

This is a thought experiment in civic economics, not a call to action: a plan any city, or any rich private citizen, could test with thirty startups and a million dollars each before betting the stadium money on it.
Duration: 10min, Publish Date: Feb 2009, Post: paulgraham.com/maybe.html

Female Founders

Y Combinator funds more female founders than VCs do, 16 of 68 companies in the current batch, 24%, up from 9% two years ago and nearly twice the industry’s 13% of series A rounds. That happened through action, not advocacy: fund founders VCs overlook, then help them beat the bias they meet elsewhere, the same approach used earlier for young founders.

Adora Cheung’s Homejoy couldn’t find a VC to lead its series A despite fast growth; a tweeted revenue graph fixed that, because steep enough growth gets funded regardless of who’s asking. Getting to parity depends on the pipeline of female programmers, still far from 50%, so the leverage point sits earlier: access to computers and, more, examples of people who’ve done it, the reason Stanford produces more startups than Yale.

This lays out a theory of where bias in startup funding actually breaks, and where fixing the founder pipeline would need to start.
Duration: 12min, Publish Date: Jan 2014, Post: paulgraham.com/ff.html

Five Founders

Five founders shaped how startups think, ranked by influence rather than fame. Steve Jobs made design central to a company’s identity, staying culturally relevant for 30 years with products people anticipate like new novels. TJ Rodgers, less famous but Silicon Valley’s best writer among CEOs, taught a candid, pragmatic mode of thinking, crystallized in his essay “High Technology Innovation: Free Markets or Government Subsidies?”

Larry and Sergey tested a hypothesis at Google: hire the smartest hackers, give them a measurable problem, and the rest resolves itself. Paul Buchheit built GMail, prototyped AdSense, wrote “Don’t be evil,” and argued that a small number of users who love you beats a large number who just like you. Sam Altman showed that a few people have enough will to get whatever they want, regardless of the odds.

This works as a quick primer on which people and phrases recur in startup advice, useful for founders or investors who want the reference points behind them.
Duration: 4min, Publish Date: Apr 2009, Post: paulgraham.com/5founders.html

Founder Control

Founders can keep majority control of the board through a series A. Ten years ago that was rare: the standard board was two founders, two VCs, one independent, and founders lost their majority either way. Mark Zuckerberg kept board control of Facebook through the series A and still has it; Mark Pincus did the same at Zynga.

Board control is not the same as total control; VCs still hold vetoes over big decisions like a sale, and votes are usually unanimous because the losing side backs down before the vote. But when opinions split, whoever holds the majority tends to prevail on what counts as the shareholders’ interest. A dozen YC-funded companies reported founders keeping board majorities after their A round, and VCs often resist conceding board seats mainly because they don’t want to look beaten in front of their partners, not because they care about the outcome itself.

This traces a shift already underway and predicts it will become the norm within a year, since the startups able to hold board control tend to be the ones that set the terms other founders and VCs follow.
Duration: 4min, Publish Date: Dec 2010, Post: paulgraham.com/control.html

Founder Mode

Airbnb’s free cash flow margin is now among the best in Silicon Valley, and Brian Chesky got there by rejecting the advice everyone gives growing founders: hire good people, give them room, stay out of the details. Followed straight, that advice nearly wrecked the company.

Running a company you founded is different from running one you were hired to manage. Chesky rebuilt his approach by studying Steve Jobs, who ran an annual retreat at Apple for the 100 people he considered most important, regardless of where they sat on the org chart, and who took skip-level meetings for granted. Founder after founder at a recent YC event described the same pattern: conventional scaling advice damaged their companies, and going around it fixed things.

There’s no book on this yet, no business-school course. What’s collected here are scattered examples of founders improvising their way past bad advice, offered as a first sketch of something not yet named.
Duration: 7min, Publish Date: Sep 2024, Post: paulgraham.com/foundermode.html

Founders’ Accents

Founders with accents so strong that people can’t understand them tend to run companies that do badly. This is not about accent as a cultural signal; plenty of Silicon Valley’s most successful founders speak with heavy accents. The problem is comprehension, not origin.

Ranking every YC company by valuation, the CEOs with strong foreign accents cluster around 100th place and below. Demo Day pitches, at two minutes thirty seconds, get memorized phoneme by phoneme, so they don’t expose the problem; office hours do, where subtle points get lost and phone calls degrade communication further, which is why YC requires founders to relocate to Silicon Valley. A founder is always selling, to employees, investors, and press who start out skeptical and won’t work to understand you.

The advice that follows: make yourself easy to understand, because startup ideas already sit close enough to bad ideas that any added friction sinks them. Aimed at founders coming to Silicon Valley from abroad, and at the entrepreneurship programs preparing them.
Duration: 4min, Publish Date: Aug 2013, Post: paulgraham.com/accents.html

Frighteningly Ambitious Startup Ideas

The biggest startup ideas are the ones that scare you off, and that fear is exactly why they’re worth naming. A new search engine, a replacement for email, a replacement for universities, internet drama delivered outside cable, the next Steve Jobs, a compiler that parallelizes code automatically, and ongoing medical diagnosis all fit the same test: hard enough to feel impossible, valuable enough to build a billion-dollar company around.

Each case comes with its own evidence: Google’s search results getting cluttered, GMail slow enough that $1000 a month for a fast replacement would be reasonable, Bill Clinton learning his arteries were 90% blocked only after symptoms appeared, Facebook starting with just Harvard undergrads before becoming the universal site.

The advice for approaching any of these: start small and specific, like a Basic interpreter for a machine with a few thousand Altair owners, and expand outward rather than announcing the big goal upfront.
Duration: 21min, Publish Date: Mar 2012, Post: paulgraham.com/ambitious.html

Hiring is Obsolete

Undergrads are undervalued, and the cost of starting a startup has dropped so low that the main expense is food and rent. Ten thousand dollars and a willingness to live on ramen is enough seed funding to test whether a 20 year old can build something people want.

Big companies buy startups for the team as much as the technology, often for two or three million dollars six months in, essentially a hiring bonus. Yahoo, Google and Microsoft were all founded by 24 year olds; Google’s own founders held off a year before accepting a CEO the VCs wanted. Nine of ten startups fail, but the one that works pays more than ten times an ordinary salary, and a failed attempt at 22 doesn’t hurt you: Yahoo’s Zod Nazem said he’d rather hire the candidate whose startup tanked.

This is aimed at ambitious college students and grad students weighing a startup against a first job, with grad school floated as a reasonable launch pad in the meantime.
Duration: 26min, Publish Date: May 2005, Post: paulgraham.com/hiring.html

How Not to Die

A 50 percent success rate for funded startups means half the founders get rich and half get nothing. The only lever that moves a company between those outcomes is refusing to die.

Y Combinator’s own track record backs this: of roughly ten startups that failed out of five funding cycles, the warning sign was always silence, not a dramatic collapse. Founders who kept shipping and staying in touch survived; official causes of death (running out of money, a co-founder bailing) mask the real one, demoralization. Blogger and Delicious took years but grew from a small core of fanatic users into successes; Octopart’s founder dropped out of grad school after a Newsweek profile made failure too humiliating to accept, and stopped being able to quit.

This is aimed straight at founders mid-startup, worried they’re failing because nothing they tried worked yet. The advice is blunt: don’t add a fallback like grad school, find the few users who love you, and keep typing.
Duration: 10min, Publish Date: Aug 2007, Post: paulgraham.com/die.html

How to Start a Startup

Getting rich from a startup is doable, not brilliant. Three things determine it: start with good people, build something customers want, spend as little money as possible. Most failures trace to one of those three, not to a missing idea.

Google’s plan was a search site that didn’t suck: index more of the web, rank by links, keep pages clean. That alone brought in a billion dollars a year. Viaweb grew slowly, hit 70 users by the end of 1996, and used that year to learn online commerce cold while competitors chased brand and headcount. Warren Buffett aside, only 5 of the Forbes 400’s top 50 hold MBAs.

For anyone weighing a startup against a salaried career: the math, the hiring test (”could you call this person an animal”), and the case for renting an apartment over an office.
Duration: 53min, Publish Date: Mar 2005, Post: paulgraham.com/start.html

Learning from Founders

Startups are most productive in their earliest phase, when Apple was just Steve Jobs and Steve Wozniak. The messiest-looking stretch of a company outproduces the buttoned-up version that comes later.

Business dresses itself up in suits, conference tables and Powerpoint, and that effort makes organizations worse, not better, the same way bolting fake spoilers onto a car slows it down; the fastest quarter mile came from a magazine stripping a car of everything the manufacturer added to make it look fast. Real programming happened in a towel at 2am in a room full of junk, not from well-dressed people at clean desks during office hours, and even founders who lived this often felt like impostors for not looking the part.

Founders at Work exists to show that first year to people who never get to see it: investors, executives, and anyone trying to tell what real productivity looks like versus what merely looks professional.
Duration: 5min, Publish Date: Jan 2007, Post: paulgraham.com/foundersatwork.html

Microsoft is Dead

Microsoft’s death happened years before anyone noticed. It ran the software world for nearly 20 years starting in the late 1980s, inheriting the monopoly IBM had held since the 1950s. By 2005, four things killed it at once.

Google took the lead in 2005, not at its August 2004 IPO; Gmail proved it could do more than search, and showed what Ajax could do on the web. Microsoft had built the key piece, XMLHttpRequest, for Outlook in the late 90s without grasping its use elsewhere, then fought Javascript while open source libraries grew over Explorer’s bugs “like a tree over barbed wire.” Broadband made the desktop optional, and OS X brought Apple back so completely that Y Combinator’s founders and Startup School’s audience all carried Macs.

Microsoft still has the cash to buy every good “Web 2.0” startup for less than Facebook alone would cost. It won’t, because it still thinks it can write software in house.
Duration: 7min, Publish Date: Apr 2007, Post: paulgraham.com/microsoft.html

Organic Startup Ideas

The best startup ideas come from something you personally want built, not something you decide, from a distance, that other people need.

Woz built the Apple I because he wanted a computer; enough other people wanted one too that Apple could sell them. Bill Gates wrote the first Basic interpreter because the Altair only ran machine language. Larry and Sergey wrote early Google for themselves. Mark Zuckerberg wasn’t starting a company when he put Harvard’s paper facebook online in 2004; neither was Woz when he started the Apple I. Both looked like toys at the time, which is usually the sign an idea is being overlooked.

This favors young founders, who lack the experience to guess what strangers need but sit closest to new technology and its fresh, fixable breakage. The advice: stop asking what’s investable and start asking what’s broken in your own daily life.
Duration: 6min, Publish Date: Apr 2010, Post: paulgraham.com/organic.html

Post-Medium Publishing

Publishers of all kinds insist consumers won’t pay for content anymore, but they never really sold content in the first place. A copy of Time costs $5 for 58 pages, 8.6 cents a page; The Economist costs $7 for 86 pages, 8.1 cents a page. Better journalism costs slightly less, which only makes sense if publishers have always been marking up paper, not prose.

iTunes gets read as proof people pay for content, but it works as a tollbooth: Apple owns the channel and dings a credit card just below the threshold of attention. Software charges real money because businesses fear pirated copies and because a Photoshop user needs Photoshop in a way no one needs a particular song. Premium cable still gets paid because broadcasting isn’t publishing at all.

Two paths remain: give the work away and profit from concerts, t-shirts, or advertising, or embody it in a physical object worth owning, like a well-made book or a lush fashion magazine. Anyone trying to guess what replaces newspapers, records, and paperback publishing gets a clear test here for telling a real opportunity from a stalling tactic.
Duration: 10min, Publish Date: Sep 2009, Post: paulgraham.com/publishing.html

Startups in 13 Sentences

Startups succeed by making a small number of users love the product, not a large number merely satisfied by it.

Thirteen rules compress to one: understand your users, since that is the dimension of wealth founders control most. Paul Buchheit’s principle anchors the list; launching fast, letting the idea evolve, and offering customer service too good to scale all serve the same end, learning what users lack before scaling to satisfy them. Cheapness (Viaweb’s habit) and “ramen profitable” status buy the time to keep iterating; 20 deals falling through taught the value of ignoring deals until they close.

This is a working checklist for founders past the idea stage: what to prioritize when cash, morale, and focus are all under pressure at once.
Duration: 8min, Publish Date: Feb 2009, Post: paulgraham.com/13sentences.html

The Airbnbs

Airbnb almost died before it worked. Brian Chesky, Joe Gebbia, and Nate Blecharczyk spent a year with no growth, funding the company on a binder full of maxed-out credit cards, and had agreed that Y Combinator was their last shot.

What kept them going was a taste of the thing itself: renting airbeds during a design conference, they and their first three guests all had a good time, proof of a new way to travel and a new way to pay rent. During YC they chased ramen profitability, $4000 a month taped to their bathroom mirror, by focusing entirely on New York City, visiting hosts in person and shooting their listings with a rented camera. Fees went from $460 to $897 to $1428 across three weeks in February 2009, in the middle of the worst recession in decades.

It reads as a case study in founder intensity: taking notes in every meeting, implementing suggestions before the next one, refusing to quit an idea that investors had already walked out on.
Duration: 6min, Publish Date: Dec 2020, Post: paulgraham.com/airbnbs.html

The Cliffs Notes - Microsoft is Dead

Microsoft being dead never meant going out of business. It meant people building new technology no longer had to think about them, the way nobody building a startup worries about SAP even though SAP makes plenty of money.

Technology companies work differently from actors or pop stars. A pop star who stops mattering can still be broke later or make a comeback; a technology company almost never comes back once it crosses into irrelevance, the way IBM did before it. Relevance can lead revenue by five or ten years, so a company can be called dead long before its balance sheet shows any trouble.

Calling Microsoft dead first meant risking being wrong: if Microsoft somehow turned itself back into something startups had to worry about, the call would look foolish. That risk was accepted deliberately, since being first to say it was the point.
Duration: 2min, Publish Date: Apr 2007, Post: paulgraham.com/cliffsnotes.html

The Fatal Pinch

Startups usually die from a specific trap: several months of runway, heavy monthly losses, flat or weak revenue growth, and a plan to close the gap with a new funding round. That plan fails more often than founders expect, because three forces stack against them the second time around: spending is higher than at the first raise, investors hold later-stage companies to higher standards, and a company that hasn’t grown now reads as a failure by default.

The trap is self-reinforcing: expecting more money makes founders slack on reaching profitability, which lowers the odds of raising it. Y Combinator’s fix is to treat every round as the last one you’ll get. If you’re already in the pinch, the probability of another raise is effectively zero, leaving three moves: shut down, cut spending (mainly headcount), or raise revenue by shifting people onto sales or taking on consultingish work, precisely defined and paid upfront rather than billed by the hour.

Written for a founder watching runway shrink, with a concrete diagnostic for whether to trim headcount or chase revenue: overhired teams should cut first, small teams should sell harder.
Duration: 9min, Publish Date: Dec 2014, Post: paulgraham.com/pinch.html

The Other Road Ahead

Web-based software runs on a server, using the browser as the whole interface, and this will out-compete desktop software for most users.

Viaweb ran this way from 1995 and grew into Yahoo Store, with 14,000 users. No installation, no operating systems, no version numbers: three programmers shipped three to five releases a day instead of one a year, cut known bugs to under ten at any time, and got the capital cost per user down to about $5. Watching users directly, one added message about the Back button raised test-drive completion from 60% to 90% and lifted revenue growth 50%.

This favors startups over incumbents like Microsoft, since writing for servers needs no OS deals, no shelf space, and lets founders launch on under $10,000, as Viaweb did with three people in an apartment.
Duration: 68min, Publish Date: Sep 2001, Post: paulgraham.com/road.html

The Pooled-Risk Company Management Company

Startup founders want two things: freedom from running the business day to day, and security against its revenue disappearing. Owning your own company gives you neither: no boss demands more than a company you depend on, and stop paying attention and the revenue stops too.

The fix already exists. A public acquisition is structurally a pooled-risk management company: it hires a manager (the acquirer) and pools your company’s fate with many others, so one dead market doesn’t sink you, the way Viaweb’s sale did for its founder. The catch is timing, not the mechanism: acquirers behave like rational risk-pooling buyers only on average, over a window of years, so you need enough runway, meaning profitability, to survive until an up cycle hits.

David Heinemeier Hansson’s advice to live off revenues isn’t wrong. It’s just not opposed to selling the company; for most founders, selling is the optimal version of exactly that plan.
Duration: 7min, Publish Date: Jul 2008, Post: paulgraham.com/prcmc.html

The Reddits

Steve and Alexis Huffman and Ohanian pitched a bad idea: ordering fast food by cellphone, before smartphones existed. It never launched. It still doesn’t exist 19 years later. But their curiosity and energy were enough to build a company around, so the offer became: we’ll fund you, just not that.

The replacement idea came from watching del.icio.us/popular, a list of most-saved links functioning as a de facto news feed. Reddit shipped a couple hundred lines of code, three weeks into the first YC batch, tested by Steve, Alexis, their YC batchmates, and multiple accounts per person. Its one difference from the incumbent, Slashdot, mattered: user submissions instead of editors, making it consistently fresher. Chris Slowe and Aaron Swartz joined soon after.

Reddit’s traffic grew slowly, then never stopped, surviving years of mediocre management after Steve left, because the product itself was close to unkillable. When Steve returned in 2015, growth resumed at a scale that showed what the company could do with someone who actually cared running it.
Duration: 6min, Publish Date: Mar 2024, Post: paulgraham.com/reddits.html

The Trouble with the Segway

Segways don’t get ridden the way their inventors hoped, and the reason is simple: standing still while a machine does the work makes a rider look smug. A motorcycle rider works just as little, but sitting astride it reads as effort; standing on a Segway doesn’t.

Trevor Blackwell built two working models to test this directly: the Segwell, a homemade Segway, and the Eunicycle, a unicycle that looks pedaled but isn’t. Riding the Eunicycle to get coffee in downtown Mountain View got him smiles. Riding the Segwell got him shouted abuse from passing cars. The company itself never ran that test: flush with funding, it built in secret with focus groups instead of real users, so nobody ever yelled an insult out a car window at it.

The fix suggested here is a version ridden foot-in-front-of-foot, skateboard style, styled like a skateboard or bicycle rather than a medical device.
Duration: 2min, Publish Date: Jul 2009, Post: paulgraham.com/segway.html

Web 2.0

Web 2.0 got a real meaning by accident, arrived at through three things that are simply the web working as it was always meant to: Ajax, democracy, and not treating users badly.

Google Maps in early 2005 proved Javascript-heavy apps could feel like desktop software, and thousands of hackers built on Ajax within months, the way none ever built on Java applets. Wikipedia and Reddit show amateurs beating edited professionals once enough of them can publish and voters can filter; editing yields 95th-percentile writing, but an unedited pool large enough beats it at the top. Craigslist and iTunes show giving users more, not less, for free is what wins, while Rupert Murdoch’s $580 million for Myspace marks how far the market has cooled since the Bubble.

None of it was invented; Google just built its entire business, search, ranking, Maps, on this pattern and got there first. Worth half an hour for anyone building or investing in web products who wants a name for what’s already working around them.
Duration: 19min, Publish Date: Nov 2005, Post: paulgraham.com/web20.html

What I’ve Learned from Users

Startups mostly break in the same handful of ways. A few thousand companies advised over a decade produce a partner who has seen nearly every failure mode within a year or two, which is why funding many early-stage companies beats funding fewer later-stage ones: more goes wrong, faster.

That knowledge doesn’t reduce to a formula. Office hours stay one-on-one because founders misjudge which of their problems matters, or don’t see the real one at all: asked “would you use this yourself,” some say no and only then find their real problem. The summer 2012 batch broke at 80 startups because the partner pool worked as an O(n^2) match, fine at 60, unworkable a third larger; sharding partners into small groups fixed it. Founders often don’t listen anyway, because good advice sounds wrong until a year later.

Colleagues matter as much as advice. Florence in the 1490s, Bell Labs, Xerox PARC: ambitious people cluster and produce more together than alone, and YC was built to manufacture that cluster on purpose.
Duration: 12min, Publish Date: Sep 2022, Post: paulgraham.com/users.html

What Startups Are Really Like

Founders who ran startups funded through Y Combinator reported the same surprises again and again, and the pattern is not new information, it’s how much more extreme everything is than a job prepares you for.

Cofounder relationships turn into something closer to marriage, tested harder than any coworker relationship. Emotional swings happen within hours, not weeks. Deals fall through routinely, investors often can’t even switch on the hardware they’ve funded, and luck matters more than the skill-focused world of hacking trains people to expect: Microsoft’s fate hinged on whether IBM demanded an exclusive DOS license. Persistence beats intelligence, launching the smallest possible version beats polishing, and changing the idea after talking to users beats sticking to the original plan.

The through-line: every unconscious expectation carried over from a job turns out wrong. That reframing is aimed at anyone about to start a company, or already running one and wondering if what they’re feeling is normal.
Duration: 28min, Publish Date: Oct 2009, Post: paulgraham.com/really.html

What the Bubble Got Right

Yahoo’s stock traded at $200 with a P/E that implied $12; even its earnings were circular, since startups funded by Yahoo-driven VC money plowed that cash straight back into Yahoo ads. The crash wiped 95% of the value, yet the company that was left still cleared $8 billion after six years, built in six years.

That residue is the point: Google reached 82 million monthly users and about $3 billion in annual revenue without ever running an ad, because low switching costs let the best product win by word of mouth alone. The same forces favor 26-year-old founders over experienced management, informal dress over suits, and startups built to be sold, like Viaweb, over startups built to grow slowly.

Useful for founders and early employees weighing options, informality, or going public early, on the argument that good ideas are starting to beat good connections.
Duration: 22min, Publish Date: Sep 2004, Post: paulgraham.com/bubble.html

Why It’s Safe for Founders to Be Nice

Nice people can win as founders, and it costs surprisingly little.

Startups grow superlinearly, so being bad at extracting money from users only scales down the Y axis, not the shape of the curve. A company making $1000 a month growing 5% a week hits $160k a month in two years; extracting only half as much per user leaves it at $80k, which is just 15 weeks behind, always, since 2 is about 1.05 to the 15th power. Growth rate matters more: shift from 5% to 6% a week while still only extracting half, and after two years you’re at $214k versus the rapacious founder’s $160k, and a year later at $4.4 million versus $2 million.

This gives founders who worry they’re too soft a concrete trade: stay generous, and put the compensating effort into growth rate instead of squeezing customers.
Duration: 5min, Publish Date: Aug 2015, Post: paulgraham.com/safe.html

Why Startup Hubs Work

Startup hubs don’t create success so much as prevent the default outcome, which is death. Most towns don’t kill startups; they just fail to save them from the failure that’s already baked in.

Two forces make the save possible, both driven by sheer density of startup people: an environment where starting a company reads as normal rather than as unemployment, and chance meetings that supply exactly what a startup needs at the moment it needs it. Sean Parker walking into Facebook’s path in 2004, Larry Page meeting Sergey Brin, a single lunch on University Ave turning up Charlie Cheever, Selina Tobaccowala, and half a dozen others. Y Combinator works by manufacturing that density on purpose, funding enough companies that someone in the batch usually has the one missing piece another needs.

The case is for anyone weighing whether location matters: not infrastructure or weather, but the number of people around you who’ve decided ambition here is normal.
Duration: 10min, Publish Date: Oct 2011, Post: paulgraham.com/hubs.html

Why There Aren’t More Googles

Startups rarely turn down acquisition offers, and the ones that do usually end up bigger than the offer. Google could have become nothing more than Yahoo’s or MSN’s search box; Facebook nearly sold when Yahoo lowballed it. Both stayed independent because acquirers underpriced them, not because their founders had some special mission.

The bigger bottleneck sits earlier, with VCs. Y Combinator gives startups $20k at the very start; VCs show up once a company is already succeeding with checks near $2 million. In between is a gap most YC startups fall into, wanting $250-500k, that VCs won’t fill. VCs chase consensus, so the ideas most likely to seem bad at first (the ones Howard Aiken’s “ram them down people’s throats” line describes) get turned away.

The fix proposed: VCs should make five $400k bets instead of one $2 million bet, skip the board seats, cut the diligence. For anyone deciding whether to fund or found something in that $200k-ish range, this explains why the money isn’t there yet and who’s likely to fill it.
Duration: 8min, Publish Date: Apr 2008, Post: paulgraham.com/googles.html

Why to Move to a Startup Hub

Startups do better in Silicon Valley than anywhere else, and the same logic applies at every scale: London to Silicon Valley is the same move as a small agricultural town to London. Whatever argument excuses staying in London also excuses never leaving the small town.

Boston proves the point inside a single country. Y Combinator alternates funding cycles between Boston and Silicon Valley every six months, yet even Boston founders get told to move west. West coast investors act faster and bolder: a VC who had met a YC founder a week earlier beat out a Boston VC who had known him for years. Boston turned down Facebook before it moved west and raised money there. Silicon Valley has pulled ahead of Boston since the 1970s, bubble included.

Real exceptions get named: family disruption, immigration cost (one Canadian startup burned six months trying to move, then gave up), industry ties like entertainment favoring New York or LA, or an investor already committed to fund you where you stand.
Duration: 8min, Publish Date: Oct 2007, Post: paulgraham.com/startuphubs.html

Why to Not Not Start a Startup

Y Combinator’s first batch, summer 2005, put eight startups through the program; four succeeded, three by acquisition, including Reddit’s merger with Infogami, and Loopt grew strong enough to be bought “in about ten minutes” if it wanted. About half those founders got rich inside two years, against an oft-cited 10 percent success rate for startups generally.

None of the failures were traumatic. Kiko’s founders worked a full year before Google Calendar squashed them, sold their software on eBay for $250,000, and immediately started Justin.TV. Sixteen reasons people give for not starting a company get examined in turn: too young, no cofounder, no idea, family to support, fear of uncertainty, parents wanting a doctor in the family. Sam Altman, funded at 19, and the claim that 70 percent of a startup’s idea usually changes within three months anchor the argument that most of these reasons don’t hold up.

This is aimed at hackers who are unsure whether to apply, weighing a startup against a cubicle job at a place like Microsoft or Google.
Duration: 33min, Publish Date: Mar 2007, Post: paulgraham.com/notnot.html

Why to Start a Startup in a Bad Economy

Startup success depends on the founders, not on economic conditions. Recessions don’t sink good companies and booms don’t save weak ones, so the calendar year is nearly irrelevant next to who’s on the team.

Technology moves independently of the stock market: Microsoft’s Basic interpreter for the Altair was exactly what 1975 needed, and waiting would have missed the window. Costs have dropped enough that running lean, the cockroach strategy, keeps a startup alive through a downturn, and customers who need to save money are actually easier to sell to. Investors get more skittish in bad times, which is the one real cost, but hackers rarely go hungry even when a startup fails.

This is aimed at anyone holding off on a startup idea until conditions improve. The advice: find the right cofounder, keep costs low, and act on the idea now rather than waiting for a better economy.
Duration: 6min, Publish Date: Oct 2008, Post: paulgraham.com/badeconomy.html

Why TV Lost

Facebook killed TV. Four forces decided it, three predictable: the internet is an open platform where anyone builds and the market picks winners, Moore’s Law kept expanding bandwidth, and piracy trained a generation to watch on a computer screen because Bittorrent and YouTube were simply more convenient than free. The fourth was harder to see coming: social applications. Teenagers wanted computers not for the machine itself but to reach friends, through social networks, multiplayer games, messaging.

Networks are hemmed in by local affiliates the way car companies are hemmed in by dealers and unions, so they add live shows to force synchronous viewing and call it cheaper production, without seeing that cheap live content should mean more volume, not less. Synchronicity and locality, the two pillars of broadcast, dissolve online: no reason to send everyone the same signal from a local source.

For anyone weighing whether local TV, network gatekeeping, or the broadcast model survives online distribution, this is the case for why it doesn’t, and why the networks producing live TV as a defense are solving the wrong problem.
Duration: 9min, Publish Date: Mar 2009, Post: paulgraham.com/convergence.html

Why Twitter is a Big Deal

Twitter is a new messaging protocol, one where you don’t specify the recipients. New protocols that actually catch on are rare: TCP/IP, SMTP, HTTP, and not much else. A protocol owned by a private company is rarer still.

Slow monetization turned into an accidental advantage. Because the founders never locked it down, Twitter feels like the older, open protocols it sits alongside; people forget a company owns it. That resemblance made it easier to spread, the way TCP/IP or SMTP spread.

This is a short, single-argument piece: one analogy, no data, no named studies. It suits anyone who wants the case for why Twitter mattered structurally, not just as a social fad, in about two minutes of reading.
Duration: 1min, Publish Date: Apr 2009, Post: paulgraham.com/twitter.html

You Weren’t Meant to Have a Boss

Big companies split into teams of about ten to make coordination possible, but each team above the individual is really a group of groups, so freedom shrinks the size of the whole tree, not just the group you sit in. A programmer’s job is to build new things, so this hits programmers hardest: legacy code, other teams’ interfaces, and approval chains cut off most of the ideas they’d otherwise try.

Working for oneself, or for one of the first ten employees of a startup, removes that constraint, the way lions in the wild seem “ten times more alive” than the same species in a zoo. Y Combinator founders arrive looking like refugees and three months later seem taller; across more than 200 startups, YC has seen no advantage from time spent at a big company first, only the ordinary benefit of being older.

The claim runs mainly toward ambitious programmers weighing a job at Google or Microsoft against starting something of their own, and toward founders deciding how large to let their company grow.
Duration: 14min, Publish Date: Mar 2008, Post: paulgraham.com/boss.html

Work, Ambition & Craft

Whole theme: 5 hours.

How to Work Hard

Great work is never effortless. It always takes natural ability, practice, and effort together; skip any one and you fall short of the best work.

Bill Gates took no days off in his twenties. Messi’s coaches remember his dedication before his talent. Wodehouse rewrote sentences ten or twenty times at 74. The limit on hours varies by person and task: five hours a day for hard writing or programming, all day during three years running a startup. Discipline toward externally imposed goals comes easily even to children; the harder skill is working toward goals nobody assigned you, which starts with a felt disgust at wasted time and ends with judging, hour to hour, whether you’re trying hard enough, doing well enough, and working on the right problem at all.

The test for what to work on is whether you find it interesting, not whether it looks impressive or lucrative. That test only works if you’re honest with yourself about your own motives and your own results.
Duration: 17min, Publish Date: Jun 2021, Post: paulgraham.com/hwh.html

How to Do Great Work

Ambitious work follows a repeatable shape: pick something you have both aptitude for and deep interest in, then push to the frontier of that field, notice the gaps in it, and chase the promising ones.

Curiosity does the driving. It picks the field, gets you to the frontier, spots the cracks, and explores them; the specifics come from painters, physicists, Einstein reading Maxwell’s equations, Copernicus and Darwin working in the shadow of a cherished but mistaken principle. Morale, small blocks of focused time, and a willingness to throw away and redo work matter as much as raw effort; the four steps are simple to state, hard to execute.

For anyone weighing whether their own ambition is realistic, this makes the case that ability and interest matter more than luck or permission, and that starting small and evolving beats planning in advance.
Duration: 63min, Publish Date: Jul 2023, Post: paulgraham.com/greatwork.html

Maker’s Schedule, Manager’s Schedule

Programmers work best in half-day blocks; a single meeting can wreck the whole stretch. Managers use time in one-hour slots and treat meetings as free; makers need long, uninterrupted units, and a meeting dropped into the middle breaks the day into two pieces too small for anything hard.

Y Combinator runs on the maker’s schedule, even though every VC operates on the manager’s. Office hours solve the conflict by clustering founder meetings at the end of the day, so they compress the schedule instead of interrupting it. Back in the 90s, working a startup meant programming from dinner to 3am, then handling “business stuff” after sleeping until 11, effectively running both schedules in one day.

Speculative meetings, the “let’s grab coffee” kind, cost a manager nothing but can cost a maker half a day’s work. This lays out the tradeoff for anyone who schedules time with programmers, writers, or founders, so they can see why declining a coffee chat isn’t rudeness.
Duration: 6min, Publish Date: Jul 2009, Post: paulgraham.com/makersschedule.html

The Other Half of “Artists Ship”

Big companies pile up checks after disasters: a supplier goes bankrupt, so every bidder must now prove solvency. Each check has a cost nobody tallies: the best supplier who skips the bid, the good one who falls just short of a threshold set high because raising it looked free.

Joel Spolsky found software under $1000 needs no approval, but above that a committee gets involved, and vetting a committee costs vendors so much they charge $50,000 for what would otherwise sell for $5000. China restricted long trading voyages before 1400 and Europe overtook it; Sarbanes-Oxley added checks meant for General Electric and gutted the US IPO market instead. Three programmers at a startup acquired by a big company went from shipping instantly to a two-week release cycle, and said they’d trade up to half the acquisition price to get instant releases back.

The pattern applies wherever someone proposes a new check without pricing its cost, whether in a company evaluating suppliers or one weighing how much approval to put between programmers and production.
Duration: 8min, Publish Date: Nov 2008, Post: paulgraham.com/artistsship.html

What to Do

One should help people, take care of the world, and make good new things. That third principle isn’t provable, but it holds because thinking is the most impressive thing humans do, and making something is the best proof of having thought well: Newton’s physics counts, and so does a painting, since a copy of a great painting, however skilled, isn’t impressive the way the original was.

For most of history the answer to how to live was wisdom, courage, honesty, temperance, justice, upholding tradition, and serving the public interest, the same for Cicero and Confucius. That answer left out making things because the landowning class asking the question had no choice in their work; Archimedes was admired as a prodigy, not held up as a model to follow. Only in recent centuries have most people had the chance to do what he did.

Raymond Chandler’s pulp fiction and Newton’s curiosity-driven physics both show the same pattern: new work often looks unpromising or impractical at first and turns out to matter anyway.
Duration: 9min, Publish Date: Mar 2025, Post: paulgraham.com/do.html

The Anatomy of Determination

Determination beats intelligence as the best predictor of startup success. Founders with Bill Gates-level smarts routinely achieve nothing, while talent matters more only in purer, single-problem domains like math than in messier ones like organized crime, or startups.

Determination breaks down into willfulness balanced by discipline, aimed by ambition, like two fingers squeezing a melon seed: squeeze unevenly and it spins off sideways. Too much will and too little discipline produces wing flutter, alternating bursts of great work and nothing, which looks like bipolar disorder from outside; too little will against rising temptation lets achievement revert to the mean, the reason Shakespeare’s Caesar feared thin men. Will seems mostly inborn, discipline can be trained, and ambition rises fastest around other ambitious people.

None of this displaces liking the work itself, the third factor alongside talent and determination. Anyone sizing up their own drive, or a founder’s, gets a working model here: which piece is undersupplied, and what narrows the gap.
Duration: 9min, Publish Date: Sep 2009, Post: paulgraham.com/determination.html

Richard Hamming: You and Your Research

Great work is not the product of luck or raw brainpower. It comes from working on important problems, with the courage and emotional commitment to stay on them, and the drive to put in one more hour than the next person.

Feynman, Shannon, and Bode show the pattern: repeated first-class output, not a single lucky strike. Bell Labs data backs the mechanics: Bode’s compound-interest argument on effort, Shannon’s average-random-code proof, Pfann and Clogston building confidence off one early success, and the “open door” correlation between exposure to other people’s problems and later relevance. Ego, personality defects, and closed doors quietly cap what able people produce.

This is aimed at anyone who already has the ability and wants to know what separates good output from work people still cite decades later: habits like Friday “great thoughts” time, working on problems with a real attack, and writing so the next person can build on it.
Duration: 67min, Publish Date: Mar 1986, Post: paulgraham.com/hamming.html

What Doesn’t Seem Like Work?

A watershed moment doesn’t announce itself; it just shows up as the point where something stops feeling like work. The heuristic: whatever seems like effort to everyone else but doesn’t to you is the thing you’re suited for.

The father’s turn came around age 12 in Pwllheli, a small Welsh seacoast town: he started working every problem in a new math textbook the day it arrived, ahead of the class, treating the problems as the reward and the chapter text as mere hints toward them. The same split shows up elsewhere: debugging, which most programmers wouldn’t volunteer for, done gladly by people who like programming; writing papers in college for friends taking classes he wasn’t enrolled in, work that was a relief for them and interesting for him.

The stranger the mismatch feels, the stronger the signal. It offers a question worth asking yourself directly, since the answer rarely surfaces on its own: what looks like work to other people but doesn’t to you?
Duration: 3min, Publish Date: Jan 2015, Post: paulgraham.com/work.html

A Project of One’s Own

A nine year old excited to get home and write his story captures the difference between working on a project of your own and ordinary work: it’s more fun and more productive, closer to skating than walking.

School routes kids toward grades and dutiful, assigned work instead of projects that are theirs, voluntary and self-directed. At Y Combinator, grades didn’t matter for picking startups; what applicants had built on their own did. The Macintosh team, Burrell Smith, Andy Hertzfeld, Bill Atkinson, Susan Kare, worked long hours by choice, the opposite of a sweatshop. Startups and open source both pay off partly because they let skaters recruit skaters and spend all their time skating.

The real obstacle isn’t ability but confidence: fear of failure stops more good projects than overconfidence ruins. What matters is recovering the careless confidence to start, then applying adult self-awareness to keep choosing projects that are actually your own.
Duration: 14min, Publish Date: Jun 2021, Post: paulgraham.com/own.html

Cities and Ambition

Cities send different messages to ambitious people, and those messages shape what you become far more than you’d expect. New York pushes wealth, Cambridge pushes intelligence, Silicon Valley pushes power, and where you live can matter more than raw ability.

The Milanese Leonardo makes the case: fifteenth-century Florence produced nearly every famous painter while Milan, just as large, produced none, though someone born there surely had equal talent. Cambridge now works the same way, with Harvard and MIT nearly adjacent and 20 more colleges around them, while New York’s ambitious professors stay second class until they start hedge funds. The Forbes 400 ratio of New York to California residents shows the shift directly: 1.45 in 1982, .83 by 2007.

This matters most for people who haven’t yet decided what kind of ambition they have, before college, when trial and error in a few different cities is how you find out which message fits.
Duration: 20min, Publish Date: May 2008, Post: paulgraham.com/cities.html

How to Do What You Love

Liking your work is not optional decoration; it’s the mechanism behind doing it well, and most people are talked out of it before they ever test it. Childhood teaches work equals pain, school reinforces it, and adults keep up the pretense that their jobs are enjoyable out of upper-middle-class convention, not truth.

The real traps are prestige and money, especially combined, as in corporate law or medicine. The test that cuts through both: would you do this work unpaid, taking a day job as a waiter to support it. Mathematicians pass; the gender-and-identity papers on Conrad don’t. Gino Lee’s rule, aim for work that makes your friends say wow, works better than chasing strangers’ opinions.

Two paths follow from here: the organic route, gradually shaping a job toward what you like, and the two-job route, working for money to fund the real work, each with its own risks. This lays out a way to test what you actually want against what you’ve only been told to want.
Duration: 24min, Publish Date: Jan 2006, Post: paulgraham.com/love.html

Disconnecting Distraction

Procrastination runs on distractions, and distractions keep getting stronger. Television took fifty years of refinement to become “visual crack”: the average American now watches four hours a day, a quarter of their waking life. The internet is worse, because it still looks like work while you do it.

The danger showed up between projects, not during them, which is why it took years to notice. Rules failed too: limiting internet use to twice a day collapsed the moment real work required more. Treating the urge like a “sentient adversary” that finds any path you leave open, the fix that held was physical separation, a second laptop across the room (the one Steve Huffman wrote Reddit on) with wifi off on the main machine.

This is a personal fix, not a system: one computer for work, one for everything else, visible enough that slipping into distraction sets off its own alarm.
Duration: 6min, Publish Date: May 2008, Post: paulgraham.com/distraction.html

Early Work

Fear of making something lame keeps most people from ever starting ambitious work, and it stops many who do start before they push through the ugly early stage.

That fear isn’t irrational; early versions of great projects often look bad even to their creators. Silicon Valley has developed a counter-custom: treat strange ideas from unknown people as a challenge to your imagination rather than a checklist of reasons they’ll fail. Y Combinator partners see thousands of ideas every six months and learn this the hard way, since missing the one that matters is obvious in hindsight. Working techniques include Hardy’s advice to slightly overexaggerate your subject’s importance, staying slightly overconfident, calling new work “just a sketch” or “a hack,” and surrounding yourself with colleagues on similar projects who can tell an ugly duckling from a baby swan.

This is for anyone starting something new who judges their own early work too harshly, whether a startup, a piece of software, or a proof.
Duration: 14min, Publish Date: Oct 2020, Post: paulgraham.com/early.html

Earnestness

Jessica Livingston and Paul Graham have one word that means more than any other when they size up founders: earnest. It means both the direction and the magnitude of someone’s motives are right, they’re doing the work for its own sake and trying as hard as they can. Most people starting companies want money and fame instead, and that substitution is what separates them from the ones who win.

Earnestness looks like naivete from outside: founders plunge into problems asking “how hard can it be?”, then discover the problem was till recently insoluble. It doesn’t seem to matter in politics, crime, gambling, patent trolling, or academic fields at the bogus end, which is itself a useful filter for choosing where to work. Henry Ford is the historical marker here: business used to resemble robbery, but making money and doing intellectually interesting work have grown steadily more aligned since.

For anyone deciding where curiosity and money actually meet, and why reporters can’t believe founders when they say they care about the problem.
Duration: 10min, Publish Date: Dec 2020, Post: paulgraham.com/earnest.html

Good and Bad Procrastination

The most impressive people procrastinate constantly, just on the right things. Working on nothing, or on something less important, is bad procrastination. Working on something more important than what you’re supposed to do is good procrastination.

Small stuff is anything with zero chance of making your obituary: shaving, laundry, replying to letters. Big work needs long uninterrupted stretches and the right mood, which is why startups run best as a couple of guys in an apartment, before hires turn them interrupt-driven. Richard Hamming’s Bell Labs question sharpens the choice: what’s the best problem you could work on, and why aren’t you?

Big problems are frightening to face directly, more so than the risk of wasted time. The fix is working obliquely, tightening the angle once underway, the way a sailboat sails closer to the wind after starting. Anyone choosing between a clean desk and an ambitious project they enjoy will recognize the tradeoff here.
Duration: 9min, Publish Date: Dec 2005, Post: paulgraham.com/procrastination.html

How to Lose Time and Money

Losing a fortune usually has nothing to do with spending. Someone with ordinary tastes struggles to blow through more than a few tens of thousands of dollars before noticing; trading derivatives can lose a million in the blink of an eye, because moving money into an investment doesn’t trip the alarm that spending it does.

Time works the same way. A day spent watching TV feels wrong within two hours. A day spent answering email produces the same nothing, but no alarm fires, because it looks like work while it’s happening. New behaviors dodge self-indulgence alarms by mimicking virtuous ones, without even being fun.

The claim: build new alarms, for money and for time, since the old ones evolved for hunter-gatherers and don’t cover fake work or bad investing. Written after selling a startup in 1998 and money.
Duration: 4min, Publish Date: Jul 2010, Post: paulgraham.com/selfindulgence.html

Six Principles for Making New Things

A design method produces contempt at first sight, and that contempt is the tell that it’s working. Simple solutions to overlooked problems, delivered informally, starting with a crude version one and iterating fast: this generated the same laughing dismissal for Viaweb, Y Combinator, the essays, and Arc.

Viaweb launched with under 10,000 lines of code and no credit card processing for its first year, while VCs insisted transaction processing was what e-commerce meant; it crushed its competitors anyway. Y Combinator looked inconsequential next to series A rounds and foot-thick term sheets. Reddit’s minimal design read as no design at all, yet it solved the real problem: surfacing what’s new and staying out of the way. Cezanne and Klee are cited as painters who worked the same way.

This is a way of recognizing your own good ideas before the market agrees with you, aimed at anyone doing creative or startup work who mistakes a hostile first reaction for a verdict.
Duration: 7min, Publish Date: Feb 2008, Post: paulgraham.com/newthings.html

Superlinear Returns

Kids and coaches teach that returns match effort put in. In business, fame, power, and knowledge, returns are superlinear instead: a product half as good gets zero customers, not half the sales.

Two mechanisms drive this: exponential growth and thresholds. Startups, bacterial cultures, and scholarship compound, doing well now makes doing well later easier, so growth rate matters more than absolute numbers, which is why Y Combinator tells founders to watch that instead. Thresholds work like a step function, a race won by a fraction of a second, an A-list with limited room in people’s heads. Fame and learning combine both. Newton’s discoveries reportedly outweighed all his contemporaries’ combined.

Since 1970, organizations damped this effect by controlling resources, colleagues, and distribution; that damping has eroded, so variation in outcomes is rising for good and bad. The advice that follows: seek work that compounds, follow curiosity over prestige, take multiple shots because luck still matters, and expect this to suit the already-excellent and the young more than everyone else.
Duration: 25min, Publish Date: Oct 2023, Post: paulgraham.com/superlinear.html

The Right Kind of Stubborn

Persistent people and obstinate people look alike from outside: both keep going after a setback. The difference sits underneath. Persistent people are attached to a goal; obstinate people are attached to their first idea about how to reach it, and that idea is usually the least informed one they’ll ever have, since it came before any real contact with the problem.

The test is what happens when you point out a flaw. The Collison brothers listen with what gets called predatory intensity, hunting for a hole in their own boat. The obstinate glaze over and repeat themselves like doctrine. Persistence turns out to need five things at once: energy, imagination, resilience, good judgement, and a goal specific enough to aim at but not so specific it rules out an adjacent discovery. Obstinacy needs none of that; kids, drunks, and fools are good at it.

Useful for anyone trying to tell, in themselves or someone they’re arguing with, whether digging in is a strength or a failure mode.
Duration: 12min, Publish Date: Jul 2024, Post: paulgraham.com/persistence.html

The Top Idea in Your Mind

What one thinks about in the shower matters more than expected: for anything hard, there’s a kind of thinking that happens involuntarily, and it only works on whatever idea currently sits on top of the mind. Most people have exactly one top idea at a time, and everything else starves for that ambient attention.

Money and disputes are the two thoughts that push out everything else, because both force themselves to the top by their nature: raising money “becomes what you think about when you take a shower,” which is why startups stall for months once fundraising starts. Newton hit the same trap after his 1672 theory of colors, writing to Oldenburg that he’d “become a slave” to defending it against Linus’s circle, five replies and fourteen pages that cost him years of ambient thought.

The fix is indirect control: you can’t steer drifting thoughts directly, but you can choose which situations let money or disputes claim that slot, and cultivate the habit of forgetting injuries fast enough that they never get there.
Duration: 6min, Publish Date: Jul 2010, Post: paulgraham.com/top.html

When To Do What You Love

Working on what interests you most usually costs money, since people pay for what they want, not what you want, unless what you want happens to be what pays.

Bill Gates loved not just programming but running a software company and writing for customers, a strange enough taste to be rewarded. Football works the same way if you’re good enough, and some people have a genuine intellectual interest in mispriced things, a taste close to greed but distinct from it. There’s an exception at the very top: Apple, Google, and Facebook all began as projects their founders did for fun, because the best startup ideas are outliers you’d overlook if you searched for them on purpose. If you’re unsure what you want, staying “upwind,” choosing math over economics, say, keeps future options open.

For anyone aiming to do great work, the calculation stops being complicated: interest is a necessary condition, not a sufficient one, and the answer is yes.
Duration: 8min, Publish Date: Sep 2024, Post: paulgraham.com/when.html

Writing & Language

Whole theme: 2 hours.

Writes and Write-Nots

In a couple decades, few people will be able to write. Writing pervades prestigious jobs while remaining fundamentally hard, because it requires clear thinking; AI now lets anyone skip straight to the output, in school and at work, so the pressure that used to force people to learn disappears.

Eminent professors caught plagiarizing usually stole mundane boilerplate, proof they were never even halfway decent writers. Leslie Lamport’s line applies: if you’re thinking without writing, you only think you’re thinking. The result is a split into writes and write-nots, which is really a split into thinks and think-nots, the same way an industrial economy split people into those who work out and those who don’t bother getting strong.

There will still be smart people, but only those who choose to be.
Duration: 3min, Publish Date: Oct 2024, Post: paulgraham.com/writes.html

The Best Essay

The best essay tells people something true and surprising about the most important topic available, and that standard collapses into an odd conclusion: the best possible essay at any moment would describe the biggest scientific discovery it was possible to make. Darwin’s 1844 essay on natural selection counts as the best essay of that year on exactly this test.

Writing well means starting with a question you have some edge on, committing to a specific wrong or incomplete answer, then rereading strictly until the gaps force better ideas. Branches get chosen by generality and novelty, not a plan; one 17-paragraph subtree got cut here, with 5 paragraphs reattached. Timelessness splits in two: permanent importance versus staying surprising, since ideas that succeed, like Darwin’s, stop teaching us anything new. Bertrand Russell’s “trial marriage” is dull now because it won and became dating.

This is for anyone who writes to think, not to perform: the payoff is a working method, breadth from reading widely, depth from solving real problems elsewhere, and the reminder that good questions matter more than effort once the questions run out.
Duration: 23min, Publish Date: Mar 2024, Post: paulgraham.com/best.html

Good Writing

Sounding good and being right are not separate qualities in writing. Sentences that flow well are more likely to express correct ideas, because you cannot easily rewrite an awkward passage into something less true: any forced change tends to be a change for the better.

The mechanism has two parts. Rewriting to fix rhythm works like shaking a bin of objects until they pack tightly: an arbitrary constraint, like a section running one line over a page, never makes the writing worse, only better. And a smoother essay is easier for the writer to reread, 50 or 100 times over, catching what “catches” the way someone sanding wood feels for a rough spot. Liars can still write beautifully, but only by half-believing false premises first; what good rhythm guarantees is internal consistency, not truth, though the two converge when the writer is honest.

This applies only to writing that develops ideas, not writing that reports ones already worked out elsewhere, like a paper following an experiment. Useful for anyone who edits their own thinking by editing their sentences.
Duration: 9min, Publish Date: May 2025, Post: paulgraham.com/goodwriting.html

Writing, Briefly

Writing well matters more than most people think, because writing does not just record ideas, it generates them. Someone who avoids writing, or writes badly, cuts themselves off from most of the ideas that writing would have produced.

The method: write a bad version fast, then rewrite it over and over, cutting anything unnecessary. Expect 80% of an essay’s ideas to arrive after you start writing, and half of the ideas you started with to turn out wrong. Read drafts aloud to catch the sentences you stumble on and the paragraphs you dread; ask friends which sentence you’ll regret most; write for a reader who won’t read as carefully as you do, the way pop songs are mixed to sound fine on a car radio. This particular piece took 67 minutes: 23 to write, 44 to rewrite.

A short, concrete checklist for anyone who writes and wants to write better, in about 30 minutes.
Duration: 3min, Publish Date: Mar 2005, Post: paulgraham.com/writing44.html

The Need to Read

Reading teaches two things at once: what a subject is, and how to write. That second part matters because writing is not a transcript of finished thoughts, it is how new thoughts get found. A good writer discovers things mid-sentence that talking them through never surfaced.

This only kicks in for ill-defined problems. Fitting two pieces of machinery together doesn’t need prose; anything solvable formally can be solved in your head. But a messy, open-ended problem almost always yields to being written about, and someone who writes poorly will struggle to think it through at all. Writing well depends on reading well, meaning read good things and extract their meaning, not just decode the words.

Audiobooks give you the ideas but skip this: hearing prose read aloud doesn’t teach you to write it. Anyone after information alone can take a shortcut. Anyone who wants ideas can’t.
Duration: 2min, Publish Date: Nov 2022, Post: paulgraham.com/read.html

Writing and Speaking

Good ideas make writing good: plain words work if you know what you’re talking about. Speaking runs on the opposite logic. Ideas matter far less than delivery, and the two skills often pull against each other.

A prewritten talk divides attention between audience and script, so ad-libbing from an outline engages people better, but it caps how long you can spend on each sentence to how long it takes to say it. Rehearsing closes that gap, the way actors do, but every hour spent practicing is an hour not spent improving the content. And audiences get dumber as they get bigger: reactions spread mob-like past around ten people, so jokes and flattery start to beat substance.

None of this makes talks useless. They work as the closest thing to a conversation with someone too busy to meet you individually, and they’re better than writing at motivating people to act, for better or worse.
Duration: 6min, Publish Date: Mar 2012, Post: paulgraham.com/speak.html

Write Like You Talk

Written language and spoken language are different, and the gap makes writing worse. Complex sentences and formal words let a reader’s attention drift, and they trick the writer into believing they’ve said more than they have.

Experts talking about hard subjects use plain sentences, no more complex than when discussing lunch, because they have less to prove and can’t afford to let words get in the way of the ideas. A sentence like “the mercurial Spaniard himself declared: after Altamira, all is decadence” (from Neil Oliver’s A History of Ancient Britain) would raise eyebrows spoken aloud to a friend, yet whole books are written that way. The fix: read a draft out loud, and replace any sentence you wouldn’t actually say. For anything unfixable sentence by sentence, explain it to a friend out loud and transcribe that instead.

Anyone who manages this lands ahead of 95 percent of writers, since so few bother to check each sentence against how they’d actually say it.
Duration: 4min, Publish Date: Oct 2015, Post: paulgraham.com/talk.html

How to Write Usefully

Useful writing should be correct, but not by hedging into vagueness; academic prose that calls everything “complex” says nothing. It should make claims as strong as possible without becoming false, then survive years of rereading before publication.

Robert Morris’s trick: don’t say anything unless you’re sure it’s worth hearing. Applied to essays, that means deleting bad sentences and abandoning whole paragraphs rather than publishing inconclusive ideas, the opposite of scientific publication bias. Importance and novelty work the same way, using yourself as a proxy for the reader: write about what you find important and what surprises you after thinking about a topic a lot, and it will likely do the same for others. Strength comes from precise qualification, saying “I think” only where certainty warrants it.

This formula, correctness plus novelty plus importance plus strength, also explains why useful essays provoke anger and misrepresentation. It closes by noting most essays remain unwritten, since cheap publishing is new and the form was rarely cultivated before the internet: Steve Wozniak’s unbuilt paper computer designs are the analogy offered for essays written without expecting an audience.
Duration: 16min, Publish Date: Feb 2020, Post: paulgraham.com/useful.html

Putting Ideas into Words

Writing about something reveals that you didn’t understand it as well as you thought. The act of choosing words is a test your ideas usually fail on the first pass; half of what ends up in an essay only occurred to the writer while writing it.

The test comes from reading your own draft as a stranger would: does it seem correct, does it seem complete. Chess players and mathematicians can form some ideas in their heads because those domains use formal language, but essay ideas resist that; two weeks on a piece and fifty rereads of a draft is normal, and would look like a disorder in conversation, which is why talking is looser than writing.

The conclusion follows directly: if writing always sharpens ideas, then no one who hasn’t written about a topic has fully formed ideas about it, and they usually can’t tell.
Duration: 4min, Publish Date: Feb 2022, Post: paulgraham.com/words.html

The Age of the Essay

Real essays don’t start with a thesis and defend it; they start with a question and follow it wherever it leads. That structure comes from Montaigne, who in 1580 called his pieces “essais,” attempts: writing something to figure it out, not to argue a side already picked.

The high school version, thesis plus supporting paragraphs plus conclusion, is a leftover of medieval law training, where rhetoric meant arguing either side well. A real essay works like the Meander river in Turkey: it winds, backtracks, and finds the most economical route by flowing toward whatever’s most interesting at each step. Interesting means surprising, something that contradicts what you thought you knew, the way it turned out no one can say who the best programmers are because you can only judge them by working with them.

This is for anyone who found English class pointless and suspected the problem was the form, not them. The Internet lets anyone publish this way now, judged by what it says rather than who wrote it.
Duration: 26min, Publish Date: Sep 2004, Post: paulgraham.com/essay.html

The List of N Things

Cosmopolitan cover lines like “7 Things He Won’t Tell You” work because a numbered list is easier to read than a real essay: the structure is handed to you up front, the points are independent, and you can skip one without losing the rest.

That same trait makes it easier to write. A talk assembled a few days before deadline, or the classic five paragraph essay (really a list of three, dressed up with “furthermore” and a conclusion), survives even if one point runs dry, because nothing depends on what came before. Restaurants work the same way: order the cheeseburger when you don’t trust the cook. The tradeoff is that a list is fixed at the title; a real essay can turn up an idea you didn’t have when you started, and a list has no room for that.

Useful for anyone who writes on a deadline or teaches beginners, and for sizing up why a headline promising “7 secrets” is baiting you to check its count.
Duration: 7min, Publish Date: Sep 2009, Post: paulgraham.com/nthings.html

The Shape of the Essay Field

Essays need to tell readers something they don’t already know, and there are only three reasons they might not: it isn’t important, the reader is obtuse, or the reader is inexperienced. Writing for smart people about important things means writing, in effect, for the young, since that’s the group least likely to already know it.

Impact equals how much an idea changes a reader’s thinking multiplied by the topic’s importance, and the two trade off against each other: big shifts in thinking usually come on smaller topics, big topics usually only shift thinking a little. The Selfish Gene is the extreme case, recasting genes rather than organisms as evolution’s protagonists. Younger readers have more room to move, so the same essay pays off more for them.

The tradeoff isn’t a conscious calculation; it’s described as a gravitational field every essayist works inside, whether they notice it or not. The takeaway is a question rather than a rule: which important things do people tend to learn late.
Duration: 4min, Publish Date: Jun 2025, Post: paulgraham.com/field.html

Write Simply

Simple sentences and ordinary words make prose easier to read, and the easier it is, the more energy a reader has left for the ideas instead of the words. Writing simply is the goal; fancy writing is a train the reader has to carry.

Complexity often hides an absence of ideas rather than concealing depth: if you say nothing simply, that becomes obvious to everyone, including you. Simple writing also travels better across time and across non-native English readers, whose grasp of ideas can outrun their grasp of the language. The method is blunt: write the first draft fast, then spend days cutting, which strips away the parts that were never the point.

This holds for anyone who writes to be understood rather than to impress, and it doubles as a working method: write fast, then cut hard.
Duration: 3min, Publish Date: Mar 2021, Post: paulgraham.com/simply.html

Ideas, Thinking & Learning

Whole theme: 3 hours.

How to Disagree

Disagreement online has a structure, and most of it sits at the bottom. Twenty years of writers writing and readers reading has given way to comment threads and blog replies, and disagreement dominates because it’s easier to write than agreement: there’s more territory left unexplored.

A hierarchy separates the levels: name-calling, ad hominem (”of course a senator says senators need a raise”), responding to tone, contradiction, counterargument, refutation, and refuting the central point, which requires quoting the actual claim and showing why it fails. Correcting a typo or a minor number while ignoring the argument is ad hominem dressed up as rigor. The levels don’t prove a reply correct, only set a ceiling on how convincing it can be.

Mean comments cluster at the bottom of that hierarchy; real refutation rarely needs meanness. Useful for anyone deciding how to respond to something they disagree with, and for reading past confident-sounding replies that never touch the actual point.
Duration: 9min, Publish Date: Mar 2008, Post: paulgraham.com/disagree.html

How to Think for Yourself

Being right isn’t enough in science, investing, startups, or essay writing. A public market investor who correctly predicts a company’s performance makes no money if everyone else predicts the same thing; the price already reflects it. The only moves worth making are the ones most people think are wrong.

Startup founders live this test directly: writing software for a tiny computer used by a few thousand hobbyists, or renting airbeds on strangers’ floors, both sounded like bad ideas to almost everyone. Independent-mindedness breaks into three parts: fastidiousness about degree of belief, resistance to being told what to think, and curiosity that keeps searching after most people stop. Schools test and rank raw intelligence constantly but never measure this, so most people misjudge which side of the line they’re on.

This is for anyone sizing up a career choice between administrative work, where being right is sufficient, and research, investing, or founding, where being right and alone in it is the only way to win.
Duration: 21min, Publish Date: Nov 2020, Post: paulgraham.com/think.html

Crazy New Ideas

Reasonable domain experts sometimes propose ideas that sound wrong, and those ideas deserve more weight than their surface plausibility suggests, not less.

A reasonable person already knows how their idea sounds, so proposing it anyway signals they know something others don’t. Everyone, including the person having the idea, applies an excessively strict filter to new thinking, which is why Copernicus published heliocentrism in 1532 and the field only came around by the mid seventeenth century. Attacks on new ideas cost attackers nothing and look clever regardless of target, while envy, factionalism, and vested careers (Darwin’s harshest critics were churchmen) push people to dismiss rather than question.

The better response is to ask why the gap exists, since either your model is wrong or theirs is, and both are worth knowing. This favors watching for reasonable experts saying implausible things now, and reading history to see what unfinished ideas looked like before they won.
Duration: 8min, Publish Date: May 2021, Post: paulgraham.com/newideas.html

Keep Your Identity Small

Political and religious arguments go nowhere for a specific reason: no threshold of expertise gates who can weigh in. Anyone with strong convictions qualifies, unlike a Javascript thread, which stays smaller because people feel they need standing to post.

The real driver is identity, not subject matter. A discussion about a battle involving living countries turns partisan fast; the same discussion about a Bronze Age battle doesn’t, because no one knows which side to take. Programming language debates degenerate the same way once people call themselves “X programmers,” which doesn’t mean the question of which language is better has no answer.

The fix follows from the mechanism: let as few things into your identity as possible, since anything you can’t think clearly about includes everything you’ve tied yourself to. Even calling yourself a scientist should commit you to nothing but following evidence, not to any conclusion.
Duration: 5min, Publish Date: Feb 2009, Post: paulgraham.com/identity.html

The Four Quadrants of Conformism

People sort into four types based on how conventional their beliefs are and how aggressively they act on them: aggressively conventional, passively conventional, passively independent, and aggressively independent. Which quadrant someone falls into depends more on personality than on the actual rules around them.

School offers the clearest evidence: tattletales enforce rules, sheep just obey them, dreamy kids barely track them, and naughty kids question them on principle. Robert George’s Princeton students who insist they would have opposed slavery are almost certainly wrong; the aggressively conventional-minded among them would have defended it, since their type responds to whatever rules prevail. Since the Enlightenment, free inquiry has replaced heresy hunting precisely to protect independent thinkers, the ones who produce new science and startups, from the conventional-minded majority. That protection has weakened since the 1980s and again with social media.

This gives independent-minded listeners, particularly scientists and founders, a framework for why their ideas draw punishment rather than argument, and why universities used to be, but may no longer be, the safest place for them.
Duration: 12min, Publish Date: Jul 2020, Post: paulgraham.com/conformism.html

What You (Want to)* Want

Free will survives being made of matter, once “want” splits into layers. You can do what you want, but not want what you want; push further and you can even change what you want to want.

That regress does not go forever. Drug addicts sometimes stop wanting to want their addiction; someone can decide to like classical music or broccoli. But add a third or fourth “want to” and change gets rare fast, the way adding 9s after a decimal point closes in on 1 without reaching it. Three or four levels in, some desire sits at the bottom that you never chose.

The fix is a formula: some statement of the form “you can’t (want to)* want what you want” is true for everyone, with n varying by person and desire. It reframes the free will question as a search for that n, not a yes-or-no puzzle.
Duration: 3min, Publish Date: Nov 2022, Post: paulgraham.com/want.html

Being a Noob

Feeling like a beginner is a sign of learning, not a sign of falling behind: the more lost you feel locally, the more you’re likely picking up globally.

The discomfort has two causes, being incompetent and doing something new, and evolution wired us to hate both the same way, because for most of human history problems repeated rather than changing. Hunter-gatherers never had to figure out cryptocurrency. That wiring now misfires the same way an appetite for scarce food misfires when food is abundant: useful once, a liability now.

The upshot: treat the noob feeling as a measure of how much new ground you’re covering, not a verdict on your competence. Moving to an unfamiliar country makes you feel dumber than staying home, yet you end up knowing more.
Duration: 2min, Publish Date: Jan 2020, Post: paulgraham.com/noob.html

Two Kinds of Judgement

Judgement splits into two kinds. In one, judging you correctly is the goal itself: court cases, grades, competitions, with appeals if you’re misjudged. In the other, judging you is only a means to something else: college admissions, hiring, investment, dating.

Picking 20 players for a national team shows why the second kind isn’t unfair. Get the 20th spot wrong and the 21st best player takes it; if ability follows a normal distribution, the gap between them is smaller than the measurement error. The selector optimized the team, not your individual score, the way a novel buyer picking a potboiler with a racy cover isn’t being unfair to a better book, just uninterested in it.

Realizing most people judging you are closer to a fickle book buyer than a careful magistrate changes how you act: less passivity, less taking rejection personally, more effort spent selling yourself. College applicants are the case made explicit.
Duration: 4min, Publish Date: Apr 2007, Post: paulgraham.com/judgement.html

A Way to Detect Bias

You can detect bias in a selection process without knowing anything about the applicants who were rejected. If a process is biased against a type of applicant, that type had to clear a higher bar to get selected, so among those who made it through, that type will outperform the rest. Check performance after the fact and the bias shows itself.

First Round Capital published exactly this by accident: among its own portfolio companies, startups with female founders outperformed the others by 63%. The catch is a valid performance measure, one not itself corrupted by the bias being tested, and roughly equal underlying ability across the groups compared. The same First Round study also excluded its biggest win, Uber, which studies of outlier-driven returns like startup investing should not do.

Data on who applies is usually locked away, but data on who gets selected is increasingly public, which means outsiders can run this check on institutions whether those institutions want them to or not.
Duration: 3min, Publish Date: Oct 2015, Post: paulgraham.com/bias.html

Alien Truth

If intelligent life exists elsewhere, it would share more than math and physics with us: also things like the principle that a controlled experiment increases belief in a hypothesis, that practice improves skill, and Occam’s razor. These “alien truths” set a target for the most general truths short of math and physics, and possibly for concepts like justice, though that one’s less certain.

The test is generosity: if an idea might plausibly apply to any intelligent being, it counts. Erdos’s “God’s book” gets extended past math to cover this wider set. AIs may eventually let us pin down the threshold precisely, by showing which properties any system we’d call intelligent has to use, but that empirical question is separate from the philosophical one of generating candidate alien truths now.

The search for alien truths is proposed as a good definition, maybe the best one available, for what philosophy should actually be doing, whether or not current philosophers do it.
Duration: 4min, Publish Date: Oct 2022, Post: paulgraham.com/alien.html

Beyond Smart

Being very smart and having important new ideas are not the same thing. Plenty of genuinely smart people in universities and research labs discover nothing. Given a choice between being smart but discovering nothing, or less smart but discovering new things, the second is obviously better, even though it doesn’t feel that way.

If intelligence isn’t the deciding factor, something else is, and much of it can be built rather than inherited. An obsessive interest in one topic is one ingredient. Independent-mindedness is another, mostly inborn but still cultivable. General techniques for working on your own projects, plus mundane factors like sleep, low stress, and good colleagues, round it out. Writing is its own case: some ideas only surface by writing essays, not by thinking first and transcribing after.

This reframes the usual story about smart people who “underachieve.” Instead of fatalism about fixed intelligence, it points to specific, learnable factors, useful for anyone trying to produce new ideas rather than just score well on them.
Duration: 8min, Publish Date: Oct 2021, Post: paulgraham.com/smart.html

Fashionable Problems

Most fields have only been explored along a narrow, fashionable path, even after years of hard work by smart people. A vast amount of space in essays, in Lisp, in venture funding, sits untouched simply because everyone keeps working on the same few problems.

Smart, imaginative people default to conservative choices about what to work on, chasing the same fashionable questions the way others chase fashionable clothes. The fix is to look at fields already declared played out and try a genuinely new angle: if it works, the payoff scales with the size of the field, since a new approach to something enormous covers enormous ground. Loving the work is the real safeguard, since it keeps you going even when a mistake convinces you the problem is too marginal to matter.

This favors people picking a research direction or a company to start, weighing a well-trodden path against an unfashionable one nobody else wants to touch.
Duration: 1min, Publish Date: Dec 2019, Post: paulgraham.com/fp.html

How to Be an Expert in a Changing World

Beliefs about a static world can safely harden with more experience. Beliefs about anything that changes cannot, and change touches almost everything except human nature. Experts go wrong by staying experts on an earlier version of the world.

A decade spent investing in early-stage startups showed a working fix: assume things change, stop trying to predict the direction of that change, and instead stay sensitive to it. Good new ideas often look bad at first, until some shift quietly makes them good; Airbnb read as a bad idea, but the founders were earnest, energetic, and independent-minded, so the bet went ahead anyway. Public, durable commitments sharpen this further: investors who passed on Google remember it for life, because a real yes-or-no bet forces rigor that casual conversation never does.

The payoff is a method, not a prediction: watch people over ideas, treat “crazy” as a compliment, and keep company with the kind of people whose discoveries will make your own beliefs obsolete first.
Duration: 6min, Publish Date: Dec 2014, Post: paulgraham.com/ecw.html

How to Do Philosophy

Philosophy has no core subject matter the way math or history does; it’s mostly a record of what individual philosophers said. The reason: ordinary words break down when pushed too far, and philosophy keeps pushing them past that point.

Aristotle set out in the Metaphysics to study the most general truths, but assumed the most noble knowledge had to be useless, so nothing checked him when he got lost in words. Wittgenstein later showed most philosophical debates trace back to confusion over language, and unclear writing about big ideas draws inexperienced but ambitious students precisely because they can’t tell confusion from depth, the same trap that pulled in a philosophy major freshman year and Sydney Shoemaker’s class on why the self doesn’t exist.

The fix proposed: ask which of the useful things we can say are most general, not which general things are true, using examples like the controlled experiment and Frankfurt’s lying/bullshitting distinction. Anyone can do this by starting specific and working up, no tenure required.
Duration: 27min, Publish Date: Sep 2007, Post: paulgraham.com/philosophy.html

How to Get New Ideas

Good ideas come from noticing anomalies: things that seem strange, missing, or broken. Standup comedy runs on the same move applied to everyday life, but the richest source is the edge of what’s known.

Knowledge grows fractally. From a distance its boundaries look smooth; get close enough to actually work at one and it turns out to be full of gaps, gaps so obvious once seen that it seems inexplicable nobody tried x or asked y. Pushing into one of those gaps can, in the best case, open up a whole new fractal bud of its own.

At three paragraphs and a couple hundred words in the original, this is closer to a note than a talk: the payoff is the reframe itself, useful for anyone doing original work and wondering where to start looking.
Duration: 1min, Publish Date: Jan 2023, Post: paulgraham.com/getideas.html

How You Know

Reading Villehardouin’s chronicle of the Fourth Crusade two or three times leaves almost nothing you can write down afterward, maybe a page. That’s not a failure of reading. What you retain isn’t the text but its effect on your mental models: of the crusades, Venice, medieval siege warfare.

A biography of Hilbert makes the point stick: Hilbert told students “a perfect formulation of a problem is already half its solution.” You can forget where you got that conviction and still carry the conviction. Reading compiles like a program: it works, but the source is gone. Because the compiling happens with whatever state your brain is in at the time, the same book run through you again at a different age compiles differently, which is the case for rereading important books rather than treating “already read” as finished.

The forgetting that felt like wasted effort turns out to be how belief and judgment actually accumulate.
Duration: 4min, Publish Date: Dec 2014, Post: paulgraham.com/know.html

Is It Worth Being Wise?

Wisdom and intelligence aren’t two names for one trait. Wisdom is a high average across situations; intelligence is a high peak in a few. A wise person usually makes the right call; a smart one solves what almost nobody else can, even if they’re erratic elsewhere.

The two used to track together, when Confucius and Socrates could treat wisdom, virtue and happiness as one package. As knowledge specializes, the curve gets more points and the average and the peaks pull apart, so a mathematician who goes to bed dissatisfied most nights, or a physicist department half on Prozac, isn’t a failure of wisdom, just the cost of work with no upper bound. A goalkeeper’s job has a perfect version; a novelist’s doesn’t.

That gap explains why smart people are so often unwise, and why chasing intelligence means indulging your own idiosyncrasies rather than disciplining them away, the opposite of how wisdom gets built. Useful for anyone doing work where the ceiling is open and the discontent that comes with it needs a name other than failure.
Duration: 20min, Publish Date: Feb 2007, Post: paulgraham.com/wisdom.html

Mark Twain: Corn-pone Opinions

A slave named Jerry preached from a woodpile that a man’s opinions trace back to where he gets his corn pone. Opinions are not reasoned out; they are absorbed from the people around us, and self-approval depends on the approval of others.

The hoopskirt shocks a village, then six months later nobody laughs at it, with no argument in between. Dinner tables in England went from six or eight wine glasses per setting to three or four the same way. Political camps split on silver or on the tariff with the same certainty on both sides, each side reading only its own party’s literature, each mistaking feeling for thought.

The account settles for calling this force conformity and naming its cause: not calculation, but the wish to stand well with the people around us. It leaves the reader with a working test for any belief: trace it back and see whether it was reasoned or merely caught from a neighbor.
Duration: 11min, Post: paulgraham.com/cornpone.html

The Bus Ticket Theory of Genius

Genius requires a third ingredient beyond natural ability and determination: an obsessive, pointless-seeming interest in something that happens to matter.

Darwin’s curiosity about natural history on the Beagle voyage looks like the same compulsion that drives bus ticket collectors, except species matter and bus tickets don’t. Ramanujan sat by the hour working out series on his slate for no reason except that he liked it. Newton poured years into alchemy and theology alongside physics; those bets failed, but the one that paid off covered the losses. The obsession works as a filter for ability, since you won’t find series interesting without the aptitude, and as a substitute for discipline, since curiosity pulls where willpower would have to push.

This favors people willing to waste time on what looks unpromising, including kids allowed to go “preposterously deep” on some random interest instead of studying only what school assigns.
Duration: 15min, Publish Date: Nov 2019, Post: paulgraham.com/genius.html

The Risk of Discovery

Newton spent as much time on alchemy and theology as on physics, and his biographies quietly edit that out. The three fields looked equally promising in his own lifetime; only hindsight makes physics look like the obvious bet.

Biographers focus on the physics because that’s what paid off, then credit Newton with unerring judgment for having pursued it. Alchemy and theology read as wastes of time now only because we know how they turned out; in Newton’s day they were, in Marc Andreessen’s phrase, “huge, if true.” He made three bets and one worked, but all three carried real risk.

The pattern applies beyond Newton: smart people aren’t separately smart and crazy, they’re placing bets whose odds aren’t visible until after the fact. Worth thirty minutes for anyone judging their own current work by whether it looks promising yet, rather than by hindsight standards no one had access to at the time.
Duration: 1min, Publish Date: Jan 2017, Post: paulgraham.com/disc.html

What Charisma Is

Charisma is not likable manner or good press. It is a trait voters can spot before a single vote is cast, one that shows up in political cartoons months ahead of election day: once cartoonists start drawing a candidate as boring, that candidate is in trouble, while being drawn as stupid or unprincipled is survivable.

The trait itself is the wish to be around people. Clinton reached over his own Secret Service agents to shake hands, visibly thrilled rather than dutiful; Kennedy kept aides on hand for the sole purpose of company because he couldn’t stand being alone. Kerry’s “patrician reserve” and Gore’s stiffness, by contrast, read as distance. Liking people makes them like you back, the same reciprocity that makes you readier to help a friend who admires you than one who’s indifferent.

This reframes charisma as a predictable, almost physical trait, like height for a basketball player, not a coachable performance. Politicians who merely remember to smile are missing the thing that can’t be faked.
Duration: 3min, Publish Date: Nov 2004, Post: paulgraham.com/recharisma.html

Why Smart People Have Bad Ideas

Good hackers often build the wrong thing first, and the fix is learning to spot it fast. A startup has to make something people will pay for; most founders discover this only after a failed first attempt.

Paul Graham and Robert Morris spent 1995 building Artix, a company to put art galleries on the Web, before art dealers turned out to hate the idea and have no money to pay for it. They abandoned it for Viaweb, software letting anyone build an online store, which succeeded. Bill Gates and Paul Allen did the same: Traf-o-data before Microsoft. Three causes recur: taking the first idea that comes to mind, staying ambivalent about making money, and picking a “safe” underpopulated market out of fear of competition rather than the one customers actually need solved.

Sorting 227 applications for the Summer Founders Program turned up the same pattern: promising people with unpromising ideas, often blogs or dating sites instead of real problems like micropayments or dying newspapers. The fix: make something people want.
Duration: 18min, Publish Date: Apr 2005, Post: paulgraham.com/bronze.html

Since starting GrowthX, I haven’t had time to read much other than Slack and PRs. The only thing I can consume these days are audiobooks. Everything else piles into a queue of articles and PDFs I keep telling myself I’ll get to.

A lot of that queue is Computer History Museum oral histories. I’ve watched plenty of the video ones, but some of the best interviews are transcript only.

So I finally coded the workflow I’d been meaning to build forever. It cleans up the transcript, splits the speakers, gives each of them their own voice, and narrates it with Deepgram Aura-2, with chapter marks so you can skip around.

The first one I wanted to listen to was Hartmut Esslinger, the founder of frog design and a design legend. After earning their into becoming Apple’s design partner by winning Apple’s Snow White competition and defined the look of Apple hardware until Jony Ive’s translucent iMacs. The Computer History Museum has a 28-page PDF, which came out as 85 minutes of audio across 20 chapters.

Audio Version of Esslinger’s Oral History 👇

Some highlights from my notes

Winning Apple’s Snow White prize by refusing the brief

Apple ran Snow White in 1982 to unify a product line that had grown by accident. Esslinger was running a small studio out of a village in the Black Forest, and got in the door through a party at a former intern’s house.

The assumption inside Apple was that the answer looked like Sony, a company he had spent a decade consulting for. He said no. “That’s Japanese. This is different. America must be different.”

What he pitched instead was “simple, white, innocent, sexy, and also a bit more radical, a new global brand.” His argument for why Apple could get away with it: “There was no real market for the product, none at all, so Apple can do what is right.”

Where he thought Apple’s own designers had gone wrong: “Olivetti never had a design language. Olivetti had a design philosophy.”

He won the competition, then told the in-house team they couldn’t implement it, because “the way you do it, you will destroy it in the first second.” Most of them left.

The two million dollar design fee was a manufacturing argument

Two million dollars a year from Apple, starting in 1982, for a studio of eight people. The highest fee ever paid to a designer at the time, and he defends it with unit economics rather than craft.

“At a million computers you save 100 bucks per computer, that’s 100 million dollars. So what is a design fee compared to that?”

The savings were engineering. Apple had been painting its plastic enclosures. “You have a piece of plastic, five dollars. You paint it, it’s 15.” So frog moved them to unpainted Lexan, coloured all the way through.

Then zero-draft moulding. Moulded parts normally need slightly angled walls so they can be pulled out of the mould, and that taper is what makes plastic look cheap. He had solved it at Sony with a collapsible core and 16 or 32 mould pieces instead of two or four, which also put more toolmakers in parallel and shipped the part sooner. Cheaper, faster, and it looked like machined steel.

The Apple account still wasn’t profitable for frog at first: “you have to prove what you can do.”

Every argument he makes for design in three hours is a manufacturing or P&L argument.

A one percent royalty, not fees, is what built frog design

In 1974, before Apple, KaVo hired frog to redesign a dental chair. “It’s all hostile; it looks like an electric chair. Why not make it friendly?”

The CEO loved the work and had no money. They were building about 20 units a year. So Esslinger asked for a royalty instead of a fee. KaVo offered one percent, he took it, and the contract ran to less than a page.

A year later, at a regional trade show, they sold 200-plus by 11am on the opening day and stopped taking orders because the factory couldn’t make them. He redesigned it for volume. It sold for more than twenty years at roughly two million a year in license fees.

That royalty, not the Apple fee, is what paid for the model shop, an early CAD system, and salaries high enough that industrial designers were paid like engineers.

A hundred ideas to Steve Jobs, five shipped

Richard Sapper designed for IBM for decades, and later did the ThinkPad. Esslinger quotes him: “I bring them a thousand ideas and they take one. And I try a hundred times and maybe they do it once. What a waste. What a waste.”

His own version at Apple: “we took a hundred ideas to Steve and we made five good ones and co-developed them.” Fifty times the throughput, same calibre of designer.

Why in-house teams go nowhere, using Motorola as the example: good designers, but one reported into engineering and another into marketing, with no connection between them, so “nobody is really held accountable for corporate vision.”

And on the companies who ask to be the next Apple: “everybody wants to be like Apple, but when you tell them what it takes — budget, perfect models, realistic simulations, absolutely no shortcuts and a huge love for people — they balk.”

How frog design doubled its team without losing itself

Winning Snow White came with a condition: an office in Silicon Valley, near Apple. frog was eight people in a converted garage in the Black Forest, baking models in the kitchen oven.

So they hired eight more and had each existing person mentor their exact counterpart for three months, role for role. Then they split the doubled team and moved half of it to Campbell, California. After that, people rotated between Germany and California every month, administration included. “I was really careful not to lose the culture.”

He treated a forced doubling as a chance to get better rather than something to survive: “the challenge is great, but we have to get better.”

Nobody was hired into a new office. The office was cloned, then moved. I’ve done org-doubling worse than this.

Thinking big and acting small

“People have to think big, but they have to act small.”

The last thing he says, after three hours.

To me, a pattern through the interview is the idea of never settling on one altitude. Pitching Apple on a new global brand, then arguing about the taper on a moulded wall, all in the same conversation.

Core

Traditional software has bugs. LLM software has behaviors.

You don't test behaviors - you observe them, measure them, and guide them. This requires a fundamentally different approach.

Three Modes of Observation

Mode 1: Introspection The model evaluates itself during generation. Like a writer reviewing their draft before hitting send. Built-in confidence scoring, explanation generation, self-critique loops. This happens at inference time, costs tokens, but provides immediate guardrails.

Mode 2: Instrumentation Automated measurement of what actually happened. Response times, token usage, embedding similarities, retrieval success rates. Like application performance monitoring but for AI quality. Runs continuously, at scale.

Mode 3: Interpretation Humans make sense of what matters. Not everything can be measured - sometimes you need human judgment on tone, helpfulness, or edge cases. Expensive but irreplaceable for subjective quality and ground truth.

What This Looks Like in Practice

During Development:

• Start with exploration, not test cases - explore what your model can do

• Capture interesting behaviors as you discover them (the good, bad, and weird)

• Write expectations, not assertions: "should be empathetic" not "must contain 'sorry'"

• Build your eval suite backwards - get it working first, then define what "good" means based on real outputs

During Deployment:

• Shadow scoring on real traffic before switching

• Gradual rollout based on confidence thresholds

• Automatic fallback to previous version if scores drop

• No binary deploy/rollback - it's a confidence dial

During Operation:

• Stream of observations, not error logs

• Distribution of scores, not pass/fail counts

• Anomaly detection, not threshold alerts

• Behavioral drift tracking, not uptime monitoring

During Improvement:

• Human labels on strategic samples, not random QA

• Disagreement cases between evaluators get priority

• Corrections become test cases automatically

• Fine-tuning happens on production-validated examples

Embracing LLM Constraints

Embrace Uncertainty: Stop pretending LLMs are deterministic. Design for confidence intervals, not binary outcomes.

Compose, Don't Compile: Evaluators should be Lego blocks - mix and match for your use case. Need safety + factuality + tone? Stack them.

Optimize for Learning: Every evaluation should make the system smarter. If it doesn't feed back into improvement, why measure it?

Human Time is Sacred: Only escalate to humans when machines disagree or confidence is low. Make their input count by turning it into reusable signals.

That survey was the most complete map of the field, and it was drawn before reasoning models and native tool calling existed. Those two changes, plus a run of 2025-2026 benchmarks, answered the question a taxonomy can’t: which of the 58 still earn their tokens.

The bigger change isn’t in the taxonomy at all. In 2024 prompting was a specialist skill. You wrote prompts if you were building an AI product. Everyone else shipped software without touching one. In 2026 every programmer writes prompts all day.

So this is a pass back through the research: what the newer studies measured, and where those numbers contradict advice that still gets repeated. Almost every claim is paper-backed or straight from provider documentation, linked inline.

What changed

• Example count. Then: more exemplars improve performance, with diminishing returns past 20. Now: it depends on the task. Classification climbs past 250, reasoning saturates at 4-125 then degrades, and one example often scores below zero.

• Zero-shot. Then: the fallback when you have no examples. Now: the default. On GSM8K, 8 exemplars bought nothing over zero-shot.

• Example labels. Then: whether demonstrations must be strictly valid is unclear. Now: settled. Random labels cost 0-5%; format and label space do the work.

• Example order. Then: order can swing accuracy from below 50% to over 90%. Now: unchanged and still unfixed. Demonstrations at the start are worth up to 6 points.

• Chain-of-thought. Then: a core reasoning technique. Now: math and symbolic only. 95% of the MMLU gain came from questions containing “=”.

• “Let’s think step by step”. Then: task-agnostic, works anywhere. Now: deprecated on reasoning models by all three vendors, at 20-80% more response time.

• Role prompting. Then: can sometimes improve accuracy on benchmarks. Now: no reliable accuracy gain on current models, and irrelevant persona detail can cost 30 points.

• Self-consistency. Then: improves arithmetic, commonsense and symbolic reasoning. Now: collapsed on frontier models, at +1.6 points for 15x the tokens.

• Tree of Thoughts. Then: a decomposition technique. Now: still works, at 5-100x the tokens. Batch jobs only.

• Temperature and top-p. Then: yours to tune. Now: reasoning models reject them outright.

• Agents and tool use. Then: not covered. Now: the loop is an API contract, not a prompt pattern.

• Writing the prompt. Then: by hand. Now: optimizers beat reinforcement learning on most tasks, if you have evals.

What prompt engineering is

The working definition is OpenAI’s: “the process of writing effective instructions for a model, such that it consistently generates content that meets your requirements.” The word doing the work is consistently. Getting one good answer out of a model is easy. Getting the same quality on the next thousand inputs is the job.

The smallest version of the whole discipline, from the Prompt Engineering Guide:

Classify the text into neutral, negative or positive.
Text: I think the vacation is okay.
Sentiment: neutral
Text: I think the food was okay.
Sentiment:

Output: neutral. One line of instruction, one worked example, one input. The model already knows what sentiment is. The example is there to stop it answering in a paragraph.

Context engineering is the wider frame. Anthropic’s line is that prompt engineering covers “methods for writing and organizing LLM instructions for optimal outcomes,” while context engineering covers curating everything else that lands in the window at inference time: retrieved documents, tool definitions, memory, history. Karpathy’s version is “filling the context window with the right information for the next step.” For a single-turn feature the prompt is most of the game. Inside an agent it’s a subset.

Prompting is part of the development cycle now

A SKILL.md for Claude is a prompt. So is the CLAUDE.md at the root of a project, the description on every tool you hand an agent, every subagent definition in a Claude Agent SDK pipeline, the goal you hand a loop, and the .prompt files in Output. None of those are chat. They ship, they get reviewed, and they break on upgrade.

Which is the part people still underrate. A 2025 migration case study found prompts stabilized for GPT-4-32k passing at only 98% on GPT-4.1 and 97.3% on GPT-4.5-preview. Prompts rot like dependencies, and nothing in CI notices unless you wrote the eval.

Anatomy, and the one part that’s a trust boundary

Instruction with a definition of done, context, delimited input, output schema. That decomposition is old news, and OpenAI’s reasoning guidance still puts it well: “avoid vague instructions by defining what counts as done.” Here it is with all four parts labelled:

# Identity                          <- role, one sentence
You are a support triage assistant for a B2B billing product.

# Instructions                      <- instruction, with a definition of done
Assign exactly one category from the list. Output the category name only,
lowercase, no punctuation. If the ticket matches none, output "other".

# Categories                        <- context the model can't infer
billing_dispute | refund_request | plan_change | access_issue | other

# Ticket                            <- the input, delimited
<ticket>
Charged twice for the October invoice, need one reversed.
</ticket>

Output: billing_dispute. Those delimiters around the ticket are doing real work. They tell the model where your instructions stop and a stranger’s text begins.

The part worth arguing about is where each piece goes. System and developer messages carry higher privilege in the instruction hierarchy OpenAI trains models to respect. Role, rules and output format go there. Untrusted content goes in user messages or tool results, never in a developer message: OpenAI’s agent safety guidance is explicit about not putting untrusted variables there. The message-role split is an access-control decision, not formatting.

Ordering is the part that stopped being portable. Anthropic says to put long documents at the top, above your query, and claims it “improves performance across all models.” Google says the same: all context first, question last. OpenAI’s general guide puts context near the end, but the GPT-4.1 guide recommends instructions both before and after a long document. Three vendors, three answers. Pick per model and measure.

Role prompting got quietly demoted

“You are an expert X” was sold as an accuracy technique. It isn’t one any more, and the vendors have moved without announcing it. OpenAI’s o3 guidance scopes role prompting to “setting the base behavior, tone and outlining the set of actions that are possible.” Anthropic’s Claude Opus 4.6 tutorial goes further: “Skip the role setting... It infers the appropriate level of expertise from the task and context you provide.”

The research is why. A Wharton study found no expert or low-knowledge persona that reliably improved GPQA Diamond across six models, and no significant gain on MMLU-Pro for five of six. Peer-reviewed work found expert personas usually neutral, and models “highly sensitive to irrelevant persona details, with performance drops of almost 30 percentage points.” A 2026 preprint measured a librarian persona dropping mean correctness on Claude Opus from 0.92 to 0.67.

Keep the role sentence for tone and scope. Don’t expect it to make the answers more correct.

The prompt-writing checklist

When a prompt underperforms, the cause is usually somewhere on this list and not in the technique.

1. State the task and what “done” looks like. Not “summarize this” but “three bullets, under 20 words each, no preamble.”

2. Say what to do, not only what to avoid. Negative instructions aren’t testable.

3. Delimit every piece of input with XML tags, Markdown headings, or """. Pick one convention and hold it.

4. Put the output format in the system or developer message, then enforce it at the API with structured outputs where the schema matters.

5. One role sentence, task-specific. No stacked adjectives.

6. Static content first, variable content last so prompt caching can hit the prefix.

7. Show it to a colleague with no context. Anthropic’s golden rule: if they’d be confused, so is the model.

The foundational techniques, re-scored

Zero-shot is the default, not the baseline

A peer-reviewed 2025 paper put Qwen2.5-72B-Instruct on GSM8K at 95.83 zero-shot with corrected answer extraction, against 95.75 with 8-shot exemplars. The examples bought nothing.

One example is the count most likely to make things worse. GPT-4o-mini on AG News scored 0.8446 F1 at zero-shot, dropped to 0.8248 at one shot, and only passed its own baseline at two (arXiv:2607.22969). GPT-5 on biomedical QA does the same thing: 0.762 zero-shot, 0.737 at one shot, 0.768 at five (arXiv:2509.04462). If you shipped a one-shot prompt without testing zero-shot, you may have paid tokens to lose accuracy.

Start zero-shot with an explicit output format. Add machinery when your evals show a gap, not in advance.

Few-shot examples are versioned config

Two to five labeled pairs, for patterns that are easier to show than to describe: a house style for summaries, an odd label taxonomy. Anthropic is the only vendor that publishes a number, and it’s “3-5 examples for best results.” OpenAI says “a handful.” Google says experiment, and to “always include few-shot examples.”

What examples teach is narrower than most people assume. Replace the gold labels in your demonstrations with random ones and accuracy falls 0-5% absolute. You can watch it happen:

This is awesome! // Negative
This is bad! // Positive
Wow that movie was rad! // Positive
What a horrible show! //

Output: Negative. Correct, despite every label above it being wrong. The examples transmitted the label space and the // format, not the answers. Strip the format and you do worse than no examples at all.

The fragility is the story. The accuracy gap between demo-selection algorithms reached 45% on MRPC. Moving demonstrations from the front of a prompt to the back flipped over 30% of QA predictions in a 2025 study, without improving correctness. Google warns that too many examples cause overfitting, and examples buried mid-prompt hit the “lost in the middle” positional problem.

Three inputs that all look like cosmetics (which examples, how many, where) move accuracy more than most model swaps do. Formatting alone is in the same range: researchers who changed nothing else measured GPT-3.5-turbo differences of up to 40% on code translation. Version the examples, diff them, cover them with evals.

Two rules with numbers behind them. Put demonstrations at the start: best position across ten model families, worth up to 6 points, and it puts the stable text where the cache can reach it. And never sort examples by label: at 1,169 shots on Clinic-150 that cost 25.7 points.

Chain-of-thought is mostly a math-formatting effect

The technique arrived twice. The first version needed eight hand-written reasoning chains per task, which is real authoring work for one prompt. Four months later, most of that gain turned out to be free: five words appended to the question, Let’s think step by step. On text-davinci-002 that took MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%, with no examples at all.

Their Figure 1 is still the clearest before-and-after:

A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?

Without the trigger the model answers 8. It halves once and stops. With it: “There are 16 balls in total. Half of the balls are golf balls. That means that there are 8 golf balls. Half of the golf balls are blue. That means that there are 4 blue golf balls.”

CoT pays when the answer needs two or more steps that depend on each other, and answering directly collapses them into one.

A meta-analysis of more than 100 papers put it at +14.2 pp on symbolic reasoning, +12.3 pp on math, and no meaningful gain on commonsense tasks. 95% of CoT’s total MMLU gain came from questions containing “=” in the question or output.

Where it works, it’s decisive. +59.9 pp on GSM8K for Llama-3.1-8B-Instruct, and Qwen-2.5-7B-Instruct went from 23.1% to 91.1% on GSM8K in the same 2026 study.

Where it doesn’t, it costs you. The same Qwen model dropped 28.7 points on HumanEval code generation. Forcing explicit steps cut o1-preview accuracy by 36.3% absolute on implicit-learning tasks. On reasoning models, prompted CoT bought small GPQA gains for 20-80% more response time, and 35-600% longer responses on non-reasoning models.

Use it for math and symbolic work on non-reasoning models. On reasoning models, use the native control instead: Gemini 3 guidance says to replace CoT prompting with thinking_level: 'high' and simplified prompts.

One thing to know before you show a reasoning trace to anyone. Causal mediation across twelve models found GPT-4 changed its answer only 30% of the time when its reasoning chain was swapped for a perturbed one. CoT changes accuracy; it does not give you an audit trail.

The advanced techniques, and what they cost

Tree of Thoughts belongs in batch jobs

Tree of Thoughts branches several candidate thoughts per step, scores them, and searches with BFS or DFS. Game of 24 with GPT-4: 74% success against 4.0% for CoT.

The appendix prices it at 0.47 for best-of-100 CoT, and the authors say it “could require 5-100 times more generated tokens than CoT” depending on the search setup. My judgment, not a paper’s: batch jobs on search-like problems, planning and constrained generation. I’ve never found a user-facing path where the latency was acceptable, and I don’t have a controlled benchmark behind that.

ReAct survived as a loop, not as a parser

ReAct interleaves Thought → Action → Observation: 71% task success on ALFWorld against 45% for action-only prompting.

In 2026 the loop is an API contract. The assistant turn carries the thought as text and the action as a structured block:

{
  "stop_reason": "tool_use",
  "role": "assistant",
  "content": [
    { "type": "text",
      "text": "I'll check the current weather in San Francisco for you." },
    { "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": { "location": "San Francisco, CA", "unit": "celsius" } }
  ]
}

You run the tool and send the observation back as a user message, keyed to the call:

{ "role": "user",
  "content": [{ "type": "tool_result",
    "tool_use_id": "toolu_01A09q90qw90lq917835lq9",
    "content": "15 degrees" }] }

Failures go back the same way with "is_error": true rather than being swallowed; the model recovers better from a visible error than from a silent one. Loop while stop_reason is tool_use, exit on end_turn. Anthropic’s docs never say “ReAct”; the mapping is structural. If you are still regexing Action: out of free text, that’s a 2023 artifact.

The Thought half is the contested part. A peer-reviewed 2025 study found performance “is minimally influenced by the interleaved reasoning trace.” StateAct scored 27.80% on WebShop with the thought step removed against ReAct’s 17.80%. When2Tool is the alarming one: adding Reason-then-Act dropped Llama-3.1-8B tool accuracy from 79.5% to 31.2%.

It cuts the other way too. Anthropic’s think tool with tuned prompting hit 0.584 on τ-bench Airline against a 0.332 baseline, beating native extended thinking at 0.412. ToolGT found structured templates beat free-form CoT which beat no thought at all.

Both results can be true. Models trained for native tool calling already know what to call, so an extra verbal layer is at best neutral. Text-formatted agents still benefit. Test with the trace and without it rather than assuming.

Self-consistency has collapsed on frontier models

Self-consistency samples several CoT paths at nonzero temperature and takes the majority answer. On PaLM-540B and GSM8K that went from 56.5 to 74.4 at 40 paths, with 5-10 paths capturing most of it. Early-stopping variants matched 40-sample accuracy on GSM8K at an average of 14.65 samples.

Now run it on a frontier model. Gemini-2.5-Pro on MATH-500 goes from 98% at one path to 99.6% at 15. A 1.6-point gain for roughly 15× the tokens.

RAG is the cheap option, not just the grounded one

The grounding case is well known: in the XRAG benchmark, GPT-4o went from 6.30% accuracy without retrieval to 75.40% with oracle retrieval.

The cost case gets skipped. The Token Tax study priced long-context prompting at ~4.50 for semantic RAG on the same manufacturing benchmark. Roughly 26× for stuffing the window instead of retrieving into it.

One failure mode to measure for: a 2026 Springer benchmark found hybrid retrieval sometimes increased hallucination in smaller models even as retrieval recall improved. Score faithfulness, not just recall.

Let the optimizer write it

The newest option here is to stop writing the prompt yourself.

GEPA runs a candidate prompt on a minibatch, captures the traces and whatever the environment said back (compiler errors, judge notes), and shows a reflection model the whole tuple so it can propose a revision. It keeps a Pareto frontier of candidates that win on at least one example rather than one global best. On GPT-4.1 Mini across six benchmarks it beat MIPROv2 by more than double: +12.19 aggregate against +5.64. Against GRPO reinforcement learning it won five of six tasks, by up to 19 points, using up to 35× fewer rollouts.

Prompt evolution now beats reinforcement learning on most tasks at a fraction of the sample cost, which was not true in 2024.

The prerequisites are the catch, and they’re the same three things evals need: a metric that scores an example, a representative dataset, and a test set you haven’t touched. Without them it optimizes noise. Across 72 optimization runs on Claude Haiku 4.5, 49% scored below zero-shot: a coin flip, driven by eval sets too small to measure with.

Three more things worth knowing before you reach for it. More data is not better: Decagon found peak performance at 20 samples, with 500 costing 2% accuracy and 75% prompt bloat. The reflection model has to be strong, since a weak one returns your seed prompt nearly unchanged. And it optimizes around bugs rather than fixing them. GEPA drove a defective GSM8K seed from 23.81% down to 13.50% while the root cause sat there untouched.

We haven’t put this in production at GrowthX. Our prompts change faster than a compile cycle would pay for, which is a statement about our release cadence rather than about GEPA.

Sampling settings: mostly stop tuning them

OpenAI. Temperature 0-2, no stated default, no top_k support. Reasoning control is reasoning_effort.

Anthropic. Temperature 0.0-1.0, default 1.0, top_k documented for advanced use only. Reasoning control is effort, the adaptive-thinking parameter.

Google Gemini. Temperature 0.0-2.0, default 1.0, top_k supported and fixed at 64 on Gemini 2.5 Pro. Reasoning control is thinkingLevel or thinkingBudget.

On classic models the advice is unchanged: 0-0.3 for extraction, classification and function calling, 0.7-1.0 for creative generation, and alter temperature or top_p “but not both.” OpenAI’s cookbook uses temperature=0 for function calling and 0.2 for structured summarization.

On reasoning models, the knobs are being taken away. Anthropic’s Claude Opus 5 / Sonnet 5 generation returns a 400 error on non-default temperature, top_p, or top_k; you steer with the adaptive-thinking effort parameter. Google’s Gemini 3 guidance recommends keeping temperature “at its default value of 1.0” and warns that lowering it can cause looping and degraded reasoning. On reasoning models there is close to nothing left to tune.

The caveat that bites people writing tests: Anthropic documents that “even with temperature of 0.0, the results will not be fully deterministic,” and Google says the same. Exact-match assertions on model output will flake.

Prompting inside an agent loop

Single-turn prompting is one shot at getting the wording right. Agent prompting has to survive dozens of iterations, each one putting tool output back into the window.

• Tool descriptions are prompt engineering. Anthropic calls them “by far the most important factor in tool performance” and asks for 3-4 sentences minimum. The evidence backs the emphasis: renaming functions to func_N significantly drops performance, and tools with edited descriptions get over ten times more usage. OpenAI’s version is the intern test: could a human use this function given only what you gave the model? Say when to use a tool and when not to; hedged instructions like "If in doubt, use [tool]" cause overtriggering.

• Make invalid states unrepresentable. Enums for finite sets, user_id not user, documented ranges. Turn on strict mode, and note it disables parallel tool calls.

• Narrow the mandate per step. OpenAI’s Atlas hardening guidance says to “avoid overly broad prompts like ‘review my emails and take whatever action is needed.’” Broad mandates compound errors across steps and widen the injection surface.

• Curate state, don’t accumulate it. Decide at each step what history and retrieval results enter the window, and filter tool output the same way.

• Tool output is untrusted input. Anything a tool fetched from a third party is attacker-controlled text arriving inside your loop.

Frameworks are checklists, not engineering

Three have published provenance. CLEAR (Leo S. Lo, Journal of Academic Librarianship, 2023) is Concise, Logical, Explicit, Adaptive, Reflective, and works as a review rubric. PROMPT Design (Sarah Hartman-Caverly, Penn State, 2024) is a fill-in-the-blanks scaffold for non-engineers; the academic AI PROMPT framework covers similar ground with more emphasis on delimiting elements and token length. PASTAS (Lo again, portal: Libraries and the Academy, 2026) was written for the agentic era, and its Tools and Safeguards items map onto the two sections above.

They help a team remember requirements. They don’t replace measuring: a CLEAR-compliant prompt that fails your evals is still a failed prompt.

Fine-tuning is still the last resort

Every provider now documents the same sequence: evals, prompting, RAG for external knowledge, fine-tuning when prompting hits a measured ceiling. OpenAI’s model-optimization guide says “the prompt engineering process may be all you need,” and its fine-tuning platform is winding down, with existing users able to create training jobs only “for the coming months.”

The thresholds worth holding to:

• Volatility. Microsoft’s split is dynamic content → RAG, stable content → fine-tuning. Hamel Husain’s version: “fine-tuning works best to learn syntax, style and rules whereas RAG works best to supply the model with context or up-to-date facts.”

• Example count. OpenAI recommends 50 well-crafted demonstrations for SFT; a peer-reviewed 2025 study puts the classification break-even near 100 labeled examples. At 20-40, Jason Liu argues you are better off with prompt caching.

• Good enough. The applied-llms.org rule: “if prompting gets you 90% of the way there, then finetuning may not be worth the investment.”

• The bill nobody quotes. Husain estimates “99% of the labor involved with fine-tuning is assembling high-quality data.” The LIMA result cuts both ways: 1,000 curated examples fine-tuned a 65B model to outputs preferred over GPT-4’s in 43% of comparisons. Quality beats volume, and curating 1,000 excellent examples is still real work.

Prompt caching is the middle path in production. Anthropic prices cache reads at 0.1× base input and AWS says caching can cut costs up to 90%. At GrowthX our source material changes weekly, so we’ve stayed on prompting plus RAG. I haven’t hit a case where a fine-tune was worth the data-assembly cost, which is a statement about our volatility rather than about fine-tuning.

Security: assume the prompt leaks and the tools lie

The documented record includes 15+ named injection incidents between 2022 and 2026, and in 2026 alone five CVE identifiers across four agent-framework incident groups. Google measured a 32% relative increase in malicious prompt-injection categories on the web between November 2025 and February 2026.

Injection. Riley Goodside demonstrated it publicly against GPT-3 in September 2022. By 2025 it was zero-click: EchoLeak (CVE-2025-32711) exfiltrated data from Microsoft 365 Copilot with no user interaction. Researchers reported successful injections in 56% of 144 tests across 36 models. Wording will not save you; the mitigations are architectural. Anthropic’s guardrail guidance says to deliver third-party content inside tool_result blocks, keep it out of system prompts, and JSON-encode it so delimiters are unambiguous. OpenAI’s agent guidance adds that defense “cannot rely only on filtering inputs. It also requires designing the system so that the impact of manipulation is constrained.” Least privilege per tool, confirmation before irreversible actions.

Leaking. Kevin Liu pulled Bing Chat’s “Sydney” instructions in February 2023. OWASP’s guidance is that “the system prompt should not be considered a secret, nor should it be used as a security control.” Keep credentials and keys out of the prompt; assume it leaks.

Jailbreaking. From DAN to Best-of-N attacks at 89% success on GPT-4o with 10,000 augmented prompts. Layered classifiers are the answer that measures: Anthropic’s Constitutional Classifiers cut jailbreak success from 86% to 4.4%. Google Cloud’s Model Armor screens prompts and responses, OpenAI’s free Moderation API covers the basics.

OpenAI calls adversarial robustness “a hard, open problem,” and Anthropic notes that even a 1% attack success rate “still represents meaningful risk.” Budget for defense in depth from day one if your agent reads email or the open web, both channels the Gemini promptware research weaponized.

Where prompts usually break

• Negative framing. “Be concise” is not testable. Anthropic recommends the positive form: instead of "Do not use markdown", write "Your response should be composed of smoothly flowing prose paragraphs."

• No output schema. It breaks parsers, and per OpenAI’s safety docs structured outputs also “eliminate freeform channels that attackers can exploit.” One constraint, two problems.

• Trusting the documented context window. Performance degrades 13.9%-85% within claimed windows, with relevant information missed in middle positions. We hit it ourselves: a vendor documented 128,000 tokens and our task started degrading well before that. Retrieve less, better.

• Technique for technique’s sake. CoT cost o1-preview 36.3 points on implicit-learning tasks, and in a 2026 medical study it cut accuracy 5.7% while few-shot examples cut it 11.9%.

• Not reading what your framework sends. Husain’s prompt inspection found LangChain’s SmartLLMChain shipping a typo ('Let'w') in its critique prompt and Guidance making 7 API calls where 2 sufficed. Log the raw request once per integration, minimum.

Evals are the job

Draft, run against a fixed test set, score, change one variable, version like code. The loop exists because output varies: even temperature 0 isn’t deterministic, so one passing run proves nothing.

This is the reasoning behind Evals Driven Development: repeated runs scored against a rubric instead of exact-match assertions. After we rewrote the rubric, the measured pass rate on our 60-query test set went from 74% to 86%. That number is ours; the method transfers. Shreya Shankar’s guidance to review at least 100 traces before designing any rubric matches our experience: rubrics written before error analysis test failures that never happen.

The checklist:

1. Collect 50-100 real inputs before writing the prompt. Include the ugly ones.

2. Simplest zero-shot prompt with an explicit output format, run 3-5 times per input.

3. Error analysis first, rubric second, written from observed failures.

4. One variable at a time: wording, examples, format, technique.

5. Version every prompt, log every run. PromptLayer’s registry documents version diffs and release labels; LangSmith and Langfuse do the same job. Ours lives in Output, where prompts and evals sit next to traces and cost tracking in the repo. Disclosure: Output is my open-source project.

6. Gate merges on eval regressions. DeepEval runs in pytest with an --official baseline; the pattern generalizes.

7. Track tokens and cost per prompt version next to quality. A quality win that doubles spend is a decision, not a detail.

Calibrate LLM judges against human labels before trusting them, and score asynchronously where you can. Synchronous judging adds seconds per request.

FAQ

How do I pick a technique? Zero-shot with an output format. Few-shot when the pattern resists description. CoT for math and symbolic work on non-reasoning models, native thinking controls on reasoning models. Self-consistency or ToT only for batch work where an eval shows a gap worth 5-100× the tokens. RAG when the knowledge is external, changes often, or is cheaper to retrieve than to stuff.

How many examples should I use? Anthropic says 3-5 and is the only vendor publishing a number. The rule that matters more: measure zero-shot and 2-5 shots before you commit, because one example often scores below none.

What temperature should I use? 0-0.3 for extraction and function calling, 0.7-1.0 for creative work, one of temperature or top_p but not both. On reasoning models, leave it alone and use reasoning_effort, effort, or thinkingLevel.

Should I tell the model it’s an expert? For tone and scope, yes, one sentence. For accuracy, no. No vendor has published a controlled ablation showing it works on a current reasoning model, and irrelevant persona detail has measured costs.

When does fine-tuning beat prompting? When prompting has a measured ceiling, the target is a stable style or format, and you have 50-100+ curated examples plus appetite for the data work. Below that, caching and RAG are cheaper.

Should I use an automatic prompt optimizer? Only if you already have an eval suite, a few hundred representative examples, and a held-out test set. With those, GEPA is the strongest thing available. Without them, half the runs come out worse than zero-shot.

Minimum security posture for a prompted feature? Assume the system prompt leaks, so keep secrets out of it. JSON-encode untrusted input and keep it out of developer messages. Least privilege per tool, a classifier on inputs, confirmation before irreversible actions.

Is prompt engineering a viable career in 2026? No, and it never was. It was a job title minted by a hype cycle. The University of Oulu study found 72 exact-title postings out of 20,662, under 0.5%.

The skill went the other way. Lightcast counted mentions rising from ~1,400 in 2023 to ~6,300 in 2024, and O’Reilly measured 456% year-over-year growth in prompt engineering content usage. It became part of the job, like knowing how to write a migration.

“Loop engineering” is the 2026 reprise of the same bs. Real skill, fake title. Learn it as part of the stack.

The 2024 version of this post aged in two years. This one will too, and the part that ages first is always the benchmark numbers rather than the method. If you run these against your own eval set and get different numbers, I want to hear about it. Mine are one team’s.

What we’ve achieved in just ~7 months since inception is remarkable. $3M+ in ARR. Profitable. Grew team to 15. Zero funding!

Here’s the backstory:

At the beginning of the year, I took a sabbatical to immerse myself in the world of large language models (LLMs). I dove deep into retrieval-augmented generation (RAG), prompting strategies, fine-tuning techniques, and more.

Fast forward to mid-September, I got introduced to Marcel Santilli. Within minutes, it was clear—he was onto something big and had the experience + ambition to deliver on it.

Marcel's pitch:

• Brands need to become publishers.

• Founders need to become influencers.

• You need strategy + experts to deliver it, but AI can make it scalable.

Having previously helped with marketing before, I immediately got it—and Marcel had already proven the model’s effectiveness. The technical and product needs for the business were exactly the kinds of things I had spent the last year+ studying.

In the past four months, we’ve spent long hours validating a roadmap, getting the right team structure, and defining the GTM. Now we have the ambitious goal of reaching $15 million in ARR by 2025. To achieve this, my top priorities are building the right infra and putting together a great technical team.

We just opened two roles for founding engineers: https://growthx.ai/careers. We are a remote-first company hiring from anywhere, as long as we can have a 4-hour timezone overlap with San Francisco. If you’re interested, don’t hesitate to reach out!

Summary

The main idea is that every single company that doesn't rely on cheating (corruption, government ties, etc.) to come to life and achieve long-term success (measured in profit/revenue) has to have at least one or more of these "powers":

1. Scale Economies

2. Network effect

3. Counter positioning

4. Branding

5. Switching costs

6. Cornered resources

7. Process

Power progression

The idea is that these powers aren't constant either, and especially in tech, things are much more dynamic than static. Just because you succeed with one of these powers doesn't mean it will sustain forever, and each power has different defensibility in the face of competition as well. He qualifies the impact and the moment you can rely on and try to chase each power based on the phases of the business:

1. Origination

2. Take-off phase

3. Stability phase

Each phase is self-explanatory, but his idea is that chasing specific barriers in the right phase of the business will build much higher competitive advantage and barriers for new players. Also, he is explicitly focused on business phases, not product life-cycle.

Powers applicable to Origination Phase

Phase where you are between the creation of the business, building everything and just launched and/or haven't achieved growth or market fit yet.

• Counter positioning

• Cornered resources

Applicable to Take-off Phase

Take-off phase is when the business starts to achieve fast growth.

• Scale economies

• Switching Costs

• Network effect

Applicable to Stability phase

The break between take-off and stability is when unit growth falls below about 30%-40% year.

• Branding

• Process power

Now to the description and examples of each power (in order of business phases):

Counter positioning (origination)

Can be identified by:

• An upstart who developed a superior business model.

• That business model's ability to successfully challenge well-entrenched incumbents.

• Steady accumulation of customers, all while incumbent remains paralyzed and unable to respond.

Ex: The new business model is superior to the incumbent's model due to lower costs and/or the ability to charge higher prices.

Barrier of entrance:

Incumbent(s) will eventually ask: "Am I better off staying the course, or adopting the new model?" If the answer is "stay off", then the barrier is high. A newcomer adopts a new, superior business model which the incumbent does not mimic due to anticipated damage to their existing business.

Counter-Positioning versus Disruptive Technologies (Clayton Christensen)

• Digital photography vs Kodak - this is a DT, but not CP.

• In-N-Out vs. McDonald's - this is CP, but not DT

• Netflix streaming vs. HBO via cable - this is both CP and DT

The case for playing humble: Cognitive Bias can play a role in deterring the incumbent. But the challenger, by its posture, may be able to influence such a move. How to attempt this? In its ascendancy, the challenger should avoid the temptation of trumpeting its superiority, instead suppressing that urge and adopting a tone of respect toward the incumbent.

Counter positioning is not exclusive. Different from other powers like Network Effect, strong counter positioning doesn't require (or create) a winner-takes-all situation.

Five stages of the incumbent when dealing with successful counter positioning from a new challenger:

1. Denial (aka, "we are not competitors")

2. Ridicule (aka, "we are better")

3. Fear

4. Anger

5. Capitulation (frequently too late)

Blockbuster execs showed all 5 stages in their public statements when facing Netflix. I personally dealt with founders on 1 and 2 many times (and always end up on 5 later).

Once market erosion becomes severe, a Counter-Positioned incumbent comes under tremendous pressure to do something; at the same time, they face great pressure to not upset the apple cart of the legacy business model. A frequent outcome of this duality? Let's call it dabbling: the incumbent puts a toe in the water, somehow, but refuses to commit in a way that meaningfully answers the challenge.

CP often underlies situations in which the following happens at the same time:

For the challenger:

• Rapid share gains

• Strong profitability (or promise of it)

For the incumbent:

• Share loss

• Inability to counter the entrant's moves

• Eventually management shake-up(s)

• Capitulation

Cornered resources (origination)

Benefit:

Cornered Resource can emerge in varied forms, offering uniquely different benefits. It might, for example, be preferential access to a valuable patent, such as that for a blockbuster drug; a required input, such as a cement producer's ownership of a nearby limestone source, or a cost-saving production manufacturing approach.

Ex: In the Pixar case, the Brain Trust produced an uncommonly appealing product—"superior deliverables"—driving demand with very attractive price/volume combinations in the form of huge box office returns.

Barrier:

Barrier in Cornered Resource is unlike anything we have encountered before. You might wonder: "Why does Pixar retain the Brain Trust?" Any one of this group would be highly sought after by other animated film companies, and yet over this period, and no doubt into the future, they have stayed with Pixar. In the case of spin casting technology, it is patent law, and in the case of cement inputs, it is property rights.

The Five Tests to validate Cornered Resource:

1. Individual/Exclusive: If a firm repeatedly acquires coveted assets at attractive terms, then the proper strategy question is, "Why are they able to do this?" Ex: if Exxon was able to persistently gain the rights to desirable properties, then understanding their path to that is key. Do relative scale allows them to develop better discovery processes? If so, their discovery processes are the Cornered Resource.

2. Non-arbitraged: cost to keep the resource should be low enough to afford consistent differential returns (ex: a movie hiring Brad Pitt might move the box office but won't necessarily 10x the cost).

3. Transferable: If a resource creates value at a single company but would fail to do so at other companies, then isolating that resource as the source of Power would entail overlooking some other essential complement beyond operational excellence.

4. Ongoing: In searching for Power, a strategist tries to isolate a causal factor that explains continued differential returns. Ex. Post-it notes patent lasted for 30 years

5. Sufficient: The final Cornered Resource test concerns completeness: for a resource to qualify as Power, it must be sufficient for continued differential returns, assuming operational excellence. Ex. George Fisher joining Kodak wasn't enough.

Switching costs (take-off phase)

Ex: SAP - 43% of users are unhappy but 85% say they will continue to pay for the product, and the sales validate their statements.

Benefit:

A company that has embedded Switching Costs for its current customers can charge higher prices than competitors for equivalent products or services. Benefit only possible when you have additional offerings to sell.

Barrier:

To offer an equivalent product, competitors must compensate customers for Switching Costs. The firm that has previously roped in the customer, then, can set or adjust prices in a way that puts their potential rival at a cost disadvantage, rendering such a challenge distinctly unattractive.

Types of switching costs:

• Financial: $ cost of investing and purchasing new solutions (ex: SAP)

• Procedural: loss of familiarity or risk of adoption of a new product (ex: "nobody was ever fired for choosing IBM")

• Relational: Affection with the provider (ex: sales team) or product/identity as a user.

Network Effect (take-off phase)

Benefit:

Network Economies occur when the value of a product to a customer is increased by the use of the product by others (Branch Out vs LinkedIn)

A company in a leadership position with Network Economies can charge higher prices than its competitors, because of the higher value as a result of more users.

Barrier:

For Network Economies, the barrier is the unattractive cost/benefit of gaining share, and this can be extremely high. In particular, the value deficit of a follower can be so large that the price discount needed to offset this is unthinkable.

Example: The value of LinkedIn's HR Solutions Suite comes from the numbers of LinkedIn users, so LinkedIn can charge more than a competing product with fewer participants.

Characteristics:

Industries exhibiting Network Economies often exhibit these attributes:

1. Winner take all: Businesses with strong Network Economies are frequently characterized by a tipping point: once a single firm achieves a certain degree of leadership, then the other firms just throw in the towel. Game over—the P&L of a challenge would just be too ugly. For example, even a company as competent and with as deep pockets as Google could not unseat Facebook with Google+.

2. Boundedness: As powerful as this Barrier is, it is bounded by the character of the network, something well-demonstrated by the continued success of both Facebook and LinkedIn. Facebook has powerful Network Economies itself but these have to do with personal not professional interactions. The boundaries of the network effects determine the boundaries of the business.

3. Decisive early product: Due to tipping point dynamics, early relative scaling is critical in developing Power. Who scales the fastest is often determined by who gets the product most right early on. Facebook's trumping of MySpace is a good example.

Scale Economies (take-off)

The quality of declining unit costs with increased business size is referred to as Scale Economies.

Benefit:

Some condition which yields material improvement in the cash flow of the Power wielder via reduced cost, enhanced pricing and/or decreased investment requirements.

Barrier:

Some obstacle which engenders in competitors an inability and/or unwillingness to engage in behaviors that might, over time, arbitrage out this benefit.

For Scale Economies, the Benefit is straightforward: lowered costs. In the case of Netflix, their lead in subscribers translated directly into lower content costs per subscriber for originals and exclusives.

Example:

Netflix paid $100M for House of Cards and their streaming business had 30M customers, then the cost per customer was three dollars and change. In this scenario, a competitor with only one million subscribers would have to ante up $100 per subscriber.

This situation creates a very difficult position for Netflix's smaller-scale streaming competitors. If they offer the same deliverable as Netflix, similar amounts of content for the same price, their P&L will suffer. If they try to remediate this by offering less content or raising prices, customers will abandon their service and they will lose market share. Such a competitive cul-de-sac is the hallmark of Power.

Branding (stability phase)

Branding as power to charge higher prices and sustain market share.

Ex: Purchased a Diamond ring at Tiffany for $16,600 and one of similar size and cut at Costco for $6,600, then asked a reputable gemologist and appraiser to assess the rings' values: Costco ring at $8,000 plus setting costs, more than $2,000 above the selling price. Assessed Tiffany ring at $10,500 plus setting costs at a non-brand-name retailer.

"You got exactly what they said you were getting. Anything that is brand-name and has developed a reputation that Tiffany has developed, they've earned it over the years for quality control. You can go there [and] you don't have to think twice about your purchase. And you pay for that."

Benefit:

A business with Branding power is able to charge a higher price for its offering due to one or both of these two reasons:

1. Affective valence - The built-up associations with the brand elicit good feelings about the offering, distinct from the objective value of the good. (Ex Tiffany's)

2. Uncertainty reduction - A customer attains "peace of mind" knowing that the branded product will be just as expected. Consider another example: Bayer aspirin. Search for aspirin on Amazon.com and you will see a 200 count of Bayer 325 mg. aspirin for $9.47 side-by-side with a 500 count of Kirkland 325 mg. aspirin for $10.93. So Bayer has a price per tablet premium of 117%. Some customers still would prefer the Bayer because of diminished uncertainty: Bayer's long history of consistency makes customers more confident that they are getting exactly what they want.

Note that the Benefit from Branding does not depend on prior ownership, as with Switching Costs.

Barrier:

A strong brand can only be created over a lengthy period of reinforcing actions (hysteresis), which itself serves as the key Barrier.

Process (stability phase)

Ex. Toyota Process (TPS)

Benefit:

A company with Process Power is able to improve product attributes and/or lower costs as a result of process improvements embedded within the organization. For example, Toyota has maintained the quality increases and cost reductions of the TPS over a span of decades; these assets do not disappear as new workers are brought in and older workers retire.

Barrier:

These process advances are difficult to replicate, and can only be achieved over a long time period of sustained evolutionary advance. This inherent speed limit in achieving the Benefit results from two factors:

1. Complexity - Returning to our example: automobile production, combined with all the logistic chains which support it, entails enormous complexity. If process improvements touch many parts of these chains, as they did with Toyota, then achieving them quickly will prove challenging, if not impossible.

2. Opacity - The development of TPS should tip us off to the long time constant inevitably faced by would-be imitators. The system was fashioned from the bottom up, over decades of trial and error. The fundamental tenets were never formally codified, and much of the organizational knowledge remained tacit, rather than explicit. It would not be an exaggeration to say that even Toyota did not have a full, top-down understanding of what they had created —it took fully fifteen years, for instance, before they were able to transfer TPS to their suppliers. GM's experience with NUMMI also implies the tacit character of this knowledge: even when Toyota wanted to illuminate their work processes, they could not entirely do so.

Innovation is the key to achieving any of the powers

Innovation is achieved by creating compelling value. There are three distinct paths to creating compelling value.

1. Capabilities-Led Compelling Value: Adobe Acrobat. Here the key capability brought to bear was Adobe's existing fluency at the intersection of software and graphics.

2. Customer-Led Compelling Value: Corning Fiber Optics. Interaction at a distance. The uncertainty in this case is technical: "Can we invent it?"

3. Competitor-Led Compelling Value: the Sony PlayStation. Sony perceived the value in immersion of 3D graphics and the gap in the competitors' offerings (Sega/Nintendo) to double down on 3D and launch PS1.

Now, I'm recharged and excited to be back!

Here are some learnings, a new project I shipped this week, and some plans for the next few months:

Aimap.today: An Experiment with Agents & Flows

I recently focused on learning about agent tooling to build an AI company researcher for personal use.

The plan: Create an agent to find funding news, identify AI-related companies, analyze their websites, build a knowledge base, write a long form article about it, and categorize them into three tiers (foundational AI tech, AI-enabled, and AI users).

I explored several agent frameworks: CrewAI, LangGraph (LangChain’s agent take), LlamaIndex's Llama-Agents (a microservice take on agents, where each agent is it’s own service), and AgentOps (very cool instrumentation tool). I also tried building flows with AirOps and n8n. In the process of fetching and enriching the data I also tried Jina Reader/SERP, You.com API, Apollo API, and Clay agent.

Ultimately, for now, I decided to take a stab at coding my own thing. Most existing frameworks focus on LLMs talking to each other, but all I needed was a workflow/pipeline to chain multiple processes together.

My current boilerplate is in Ruby, and I wasn't satisfied with the existing packages for API clients like Claude and OpenAI. So, I coded these from scratch as well. The whole project, including study time, took about two weeks.

👉 The result is: https://aimap.today . Please, check it out and let me know what you think.

I built this for myself and plan to add new companies weekly. There's more I want to do, including a weekly digest for newly researched companies. If you sign-up you’ll be notified when I finish the weekly digest part.

My next step is to explore more out-of-the-box platforms like AirOps and n8n, and try to covert my little workflow to these tools.

Code-wise the pipeline currently looks like this:

Joined Techstars for the First San Francisco class

I'm thrilled to share that I’ll be helping Techstars’ first San Francisco class as an EIR (Entrepreneur in Residence). I've been following my friend and former boss, Neal, as he ran Techstars Chicago and Oakland. When he mentioned the new San Francisco class, I jumped at the chance to participate.

The cohort is impressive, featuring many interesting AI-enabled businesses. I'm excited to work with all the founders. You can view the full cohort here: https://www.techstars.com/newsroom/techstars-announces-inaugural-class-of-san-francisco-accelerator

It's great to be back. Building aimap.today has been an a cool learning experience, and I'm eager to continue improving and expanding it.

Joining Techstars as an EIR is another exciting opportunity I'm looking forward to.

I also may be joining a team on a new project soon, but I'll save that for future updates.

Going forward, my goal is to resume weekly updates.

Three types of play

Started considering which spaces are interesting now. I think this take from Hamilton Helmer is a good one: Three types of play:

1. The tech play: example of semiconductors, you get Intel and etc.

2. The first-tier play: companies that wouldn't exist before semiconductors (Microsoft)

3. The usage play: car companies with tons of semiconductors. They already existed, just got better.

The whole video is worth watching, but heres’ the time-stamp about AI.

I'm personally more interested in the second play. And finding these companies early and identifying which ones are already in a good direction is so hard.

Brainstorm exercise

A good place to start could be looking at common roles and how they will be impacted and can be augmented or even replaced.

A programmer's job is already seriously impacted, customer support even more so. Same for SDRs. Some other roles, like localization, have already been pretty much replaced. These highly impacted roles are the things I'm most interested in.

So, just as a thought exercise toying with Python + Langchain, I wrote a 4-step chain with Claude 3.5 to do following:

1. Brainstorm departments and roles common across any vertical

2. List the most common tasks in those roles

3. Brainstorm ideas for how each could be augmented or automated

4. (Rank them by difficulty and confidence, but this was just for nudging the model)

I only spent a couple of hours and $3 in credits, but the results turned out better than I was expecting.

There's obviously some nonsense and some things that are regular ML models more than LLMs in the output (my fault, I should have optimized the last step of the chain better), but a lot of solid ones that I think haven't happened yet, and some others in the “augmentation” category fully in progress like Figma, Intercom, Cursor, GitHub Copilot, etc. If you squint and look at the roles with the best ideas, you can see Piper, Fin, Harvey, Wisq, Devin, 11x.ai, etc

You can see the whole list here:

https://docs.google.com/spreadsheets/d/1vpXzfUFNhIZJp9CEmiZqGKaVrP_Xx8McMZZlLCxiHNg/edit?gid=0#gid=0

Next step, I guess, is to start collecting companies and maybe have a little agent help me with the research.

—

I haven't had time to think through the whole list yet, but here...Some of the ones that stood out that the ideas are mostly very doable, or already happening:

Traditional machine learning

When you move beyond simply calling APIs, you'll encounter the traditional ML stack and terminology. For instance, if you need to adjust your embeddings and choose a different model, you'll need a solid grasp of metrics like BPB, BPC, Perplexity, and so on. The same goes for evaluating production data.

I also prefer not to have components in a stack that I don't understand at a high level, even if I don't have to work with them directly (for example, if I'm an EM on the product side or a product manager).

For ML theory I’m currently half way through this course. It may be a few years old, but I appreciate how practical and real-world focused it is. If anyone has other suggestions, I'm very open to them.

I can't recall the exact source of this slide (I think it was from Hamel Hussain’s talk on AI Engineer conf this month) , but I believe it provides a reasonable overview - and my goal is to have at least a basic understanding of everything under the horizontal line.

Evaluations

Last week, I began studying evaluations more seriously for two reasons. First, I'm working on understanding the various stages of the development cycle. Second, I need to establish a better workflow for iterating on our AI bot at Canopy.

For theory around evals I really liked Chip’s third chapter of her WIP book. This post about Evaluations from Hamel Husain is also so good, and this one as well.

This is would be the ideal cycle for me:

At Canopy, our tech stack is entirely Ruby, and our AI Assistant was built using direct API calls to OpenAI. While I did implement some observability, I still need a more robust way to monitor the quality of answers in production and a solid test suite that I can run when updating the model or modifying the prompts.

Ideally, I would have automated tests similar to regular test units that I can run whenever I'm iterating on prompts, models, or content format for the rag. Once things are deployed, I'd like to have comparable quality metrics for the production data, with alerts if quality drops below certain thresholds and proper visibility.

There are quite a few LLM-focused evaluation tools available, and many of them are really solid. I don't mean to dismiss their work, but for my specific context, here's my current assessment:

1. They do too much

   1. Most tools attempt to offer solutions for all four stages of my diagram, but these companies are often small, seed-level teams building in a rapidly evolving space. Many of these products feel either half-baked or are still trying to figure out their business model. I'd prefer a tool that focuses solely on the two evaluation parts (dev & production) and doesn't try to handle the prototyping and deployment aspects.

2. Their business models are still unclear

   1. Many tools are in the typical early-stage B2B mode, lacking a product-led path and aiming to lock users into annual contracts with case-by-case pricing. I understand this is important in a new market with early-stage products, but I'd love to see more product-led options (especially for Canopy's context, a small team where the bot isn't their core feature yet).

3. Vendor lock in / dependency risk

   1. To enable tracing, many tools heavily wrap around API clients for services like OpenAI, and some don't even offer public API versions. Tools that offer "solutions" for the deployment part often proxy calls under their own API, so instead of calling OpenAI directly, you call their API. This introduces a third-party proxy as a potential failure point, in addition to OpenAI/Anthropic/etc., and it's built by an understaffed seed company.

4. Pretty much no solutions for non-Python/JavaScript stacks

   1. Most niceties for automatic logging/tracing and basic metrics like latency come from their SDK, which are often limited to Python/JS. While understandable given the early stage, most tools also don't offer a public API. For more elaborate setups, a reasonable workflow could be to follow the traditional ML path of using Python for the stack and wrapping it as an API for other languages to call, instead of relying on a proxy maintained by a tiny company in a crowded space. It's a bit of a bummer because for AI products that only need LLMs, it's an overcomplication in the stack. You'll definitely need query augmentation, retries, rate-limiting controls, and other things a product-level software already handles, and now you'll need to duplicate that in the Python stack too. It's one thing when you're making an API around a model you control, but it's another when you're making an API around another API.

If you're already using a Python or JS stack, you have fewer things to worry about. However, for Canopy (Ruby stack), the situation is a bit different. I think I'll go with the following approach:

• First: Find a tool that has a public API support for sending run data for production, build a small Ruby client for that, and do production level monitoring only (similar to how’d use a service like Sentry). Ignore all the other 3 stages of the Experiment and Deploy part of these tools.

  • Second:

    • Have a good unit test coverage for the direct calls to OpenAI with VCR and reimplement some of the basic eval metrics that relevant to my context like Answer Relevance, Context Recall, or Context Relevancy.

    • Or use something like Promptfoo, Ragas or GuardRails for the local unit tests.

I haven't decided which product to use yet, but reading through their documentation, examining the code of their SDKs, and exploring their offerings has been a good learning process already.

In no particular order, here's the current list of evaluation products/projects that I'm aware of (although I haven't had a chance to thoroughly check all of them yet):

• https://mlflow.org

• https://www.vellum.ai/

• https://athina.ai

• https://www.patronus.ai

• https://wandb.ai/site/weave

• https://autochain.forethought.ai/

• https://ragas.io/

• https://github.com/agenta-ai/agenta

• https://klu.ai/

• https://www.braintrustdata.com/

• https://humanloop.com/

• https://www.parea.ai/

• https://docs.llamaindex.ai/en/stable/understanding/evaluating/evaluating/

• https://www.langchain.com/langsmith

• https://github.com/langfuse/langfuse

• https://www.rungalileo.io/

• https://www.guardrailsai.com/

• https://deepchecks.com

• https://arize.com

• https://github.com/UKGovernmentBEIS/inspect_ai

If you're using a tool that you really like or if you have any feedback or a different perspective on the market, I'd love to hear your thoughts.

While I haven't done much hands-on AI engineering this last two weeks, I've been keeping up with theory. Here's some of the notes:

Daily Prompting Paper

For a while now, I've been checking Arxiv daily for new papers on Prompt Engineering. It's a topic that's usually easy to understand, even in academic papers, so great way to kill time between tasks.

Many poor results in AI come from inadequate prompting skills + techniques like prompt chaining and retries, weren't practical before, but now make sense.

In June, this excellent paper was published. I spent the last couple of weeks reading it thoroughly, along with many of its citations. I've compiled some notes here, and I'm also planning to create a better structured resource with more examples and additional notes I didn't include.

Meetups and Conferences

I've been attending several meetups and conferences, including AIQCon yesterday. It's interesting to see the growing focus on AI Engineering (meaning implementing GenAI in products). Last year, there a lot of interest in LLMs themselves, but now the emphasis is on making LLMs perform well in real-world applications.

Key topics include:

• RAG (Retrieval-Augmented Generation): Techniques to improving embeddings, content augmentation, and advanced techniques like GraphRAG.

• Evaluation: Setting up good test suites, deciding what to test, and determining who should write the tests.

• No fine-tuning: The consensus is to focus on prompting, RAG, and evaluation before considering fine-tuning. Lot of pain and scar from fine-tuning already.

While some events feel like they have a high noise-to-signal ratio, I always learn a lot as a newcomer. Also, it’s such a great time to live in SF. For future conferences, I plan to research the participants and their topics more thoroughly beforehand.

A useful resource from yesterday's conference: https://huyenchip.com/llama-police.html by Chip Huyen (well worth following on X).

Books

On the topic of learning more theory, I'm currently reading two books:

1. Programming Machine Learning

https://www.amazon.com/Programming-Machine-Learning-Zero-Deep/dp/1680506609

I first read this when it was released, but I've forgotten most of it. Although it's a bit outdated and pre-LLM, I like the author's writting. Given how close AI Engineering is to traditional ML, I want to revisit this material.

2. Neural Nets mini-course by 3Blue1Brown

Another basic entry level material I was revisiting this week is this mini-course on neural nets: https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

2. Oreilly’s AI Engineering book

https://www.oreilly.com/library/view/ai-engineering/9781098166298/cover.html

This book was recommended by a close person, and I attended a talk by the author yesterday - it was one of my favorites, despite being too short. Only three chapters are available now, so I'll keep an eye out for future updates.

Context: In May I left my full-time job to start a 3-month LLM learning cycle by building a few products of my own. I’m documenting my journey here. Every week I post a recap of how the week went, things I learned, etc.

Had a bit of a shorter week. I went on a camping trip to Yosemite and had no service for 3 days. Despite this, still managed to pack 3 days of 10 hours.

Here's some of the progress and learnings:

Canopy AI: JSON vs XML and streaming

I'm still helping maintain the AI Assistant I built for Canopy. It has the usual Chat experience where a user asks a question, we do a RAG search and layer a bunch of things on top.

One frustrating aspect has been GPT-4's unreliability in returning JSON. OpenAI's Completion API has a JSON mode that ensures valid JSON output, but this option isn't available on the Assistant API when using the File Retrieval tool (their built-in RAG).

Although OpenAI's built-in RAG gives me better results compared to Pinecone and PG Vector, the trade-off for Canopy's use case was losing the JSON mode.

Canopy's assistant does more than just RAG the content. It runs categorization, sensitivity checks, confidence checks, and other tasks that might trigger subsequent LLM calls or be needed in the user interface. These constraints and extra information are metadata that I used to include in a JSON (making a single API call for speed and cost).

The problem with JSON is streaming. Ideally, you want to start rendering the message as you receive the response. In the JSON, I had a markdown answer as one of the values. I'd buffer the JSON until the right value starts streaming and relay that to the user interface, stopping when the value ends. A better solution would be to have the markdown first, followed by the metadata in a structured data format (JSON, XML, etc.). After a full day of prompt engineering, I couldn't consistently get GPT-4 to return the metadata in this format after the markdown message – until GPT-4o.

Last week, I spent a day migrating from the JSON format to a structure with markdown first, followed by the metadata inside XML. Anthropic uses XML in their examples, so I've been using this tip with OpenAI as well (makes sense if you think about it, it’s easier for transformers to handle words than curly braces).

The end format looks like this:

Markdown answer…
```xml
<metadata>
…
</metadata>
```

Incorporating the ``` is another trick that took me a while to learn. When you can't use JSON mode, OpenAI will often add wrappers like these. Instead of trying to remove them in the prompt or handling failures in the app, including "shots" (examples) with the ``` as part of the answer will guide the model to always return it.

This simplified the code on our app side and made the metadata correct nearly 100% of the time. The next step is to auto-retry on the rare cases when it fails, which I haven't implemented yet.

The New Product

So far, in these 3 weeks, I've spent around 40 hours on the new product, with a significant portion of that time dedicated to boilerplate tasks that I won't need to repeat for future products, which is a positive.

While most of the AI work was already done, I couldn't resist adding some extra features at the last minute with GPT-4o. Scope creep? Why not!

I finally have a name – but I’ll share it once the landing page is up. Until now I had it under a code name, but now I have the final name, domain, and a marketing landing page 90% done.

Selecting the domain and name included initial SEO work. I think it’s BS that domains are not relevant for SEO anymore. If you are a nobody in page authority, I think it’s a good idea to make the domains very literal, so that’s how I’m starting: a short name followed by something literal.

In the 2 days I had after spending a day on Canopy, I worked on:

• Feature work:

  • Refining the AI features and adding some extra parallel LLM calls.

  • Finished an Insights page (I hate styling Heatmaps).

  • Added dark mode support while stuck in traffic driving to Yosemite.

  • Started with a Telegram bot.

• Marketing

  • Finished a landing page and registered the domain.

  • SEO: Research to find keywords I could try to rank for and figure out a strategy for evergreen content (I'll need more time here).

Before sharing the product with everyone, I still have to polish some final details, like setting up the Stripe account, deploying to the final servers, etc.

Timeline-wise, I'm thinking I need:

• 1 more full week for polishing, details, and putting up the waitlist (including boring things like Terms of Service).

• About 3 days for the initial marketing work, including backlink work, evergreen SEO-focused pages, and some community-based distribution.

Then, I'm switching attention to the second product that I want for myself (while keeping 25% of my time on this one).

Once the product is live with users, I plan to share some of the lessons more openly by writing specific posts about certain things.

For example, on this project, I've been using a mix of multiple parallel LLM calls and OpenAI batch processing. These two things could be interesting as standalone posts.

I also hope to carve out time to write separate posts about some of the product and marketing aspects whenever I can.

Week 4 Plan

This week, I'll be splitting my work into three parts:

1 day for Canopy

2 days on New Product

3 days helping a friend with his startup (I usually work a 6-day week anyway)

I'm avoiding doing any kind of external work, but in this case, I made an exception.

It's a fixed-scope project on something I'm quite familiar with, plus it's an opportunity to learn some new React on a real product designed by great programmers. On top of that, what this friend is building is an impressive product in an interesting domain with a capable team, so I'll help whenever I can. I’m estimating probably 14 days on this (let’s see how off I’ll be this time).

This will delay my LLM learning cycle plan a bit, but it's a good trade-off.

I'll continue to share my updates here regardless.

Previously, people were using the inferior free GPT-3.5. With GPT-4o, there's no excuse for uninformed takes flooding social media and YouTube. OpenAI's terrible ChatGPT UI and closed-source GPT store are still problematic. At least Anthropic is trying to teach better prompt writing (with more effort than OpenAI).

But back to my rant...

Many users do not try to tune the system, ask poorly written questions, get unsatisfactory results, and then complain online. We're 2 years into this cycle - it's time to wise up.

By now, we should all know that these systems are few-shot predictive models that respect the chain of command. Yet users provide neither the few-shot examples nor the command it should follow. It's no wonder ChatGPT-written resumes and landing pages sound like professional BS with poor English.

To get good results, you need to:

1. Give it examples

2. Understand the chain of command

GPTs can handle Zero-shot and Few-shot (but not really)

• Zero-shot learning: performing new tasks using broad pre-training

• One-shot and few-shot learning: generalizing from 1-to-many examples to perform a task on new inputs

While GPT should handle both, its zero-shot capabilities aren't there yet beyond toying around. Always provide 2-10 examples of what you want - otherwise, you'll get predictions based on general knowledge, which means bad results.

The GPT Chain of Command

1. OpenAI and other providers set the basic rules and capabilities.

2. Developers can customize these settings for specific tasks.

3. End-user requests go through the customized settings.

For non-developers, the best way to leverage the chain of command is to create a custom GPT with a system message specifying your requirements, along with necessary examples.

If you're coding against the APIs directly, you have more options beyond just system messages, such as fine-tuning. However, in the context of ChatGPT, system messages are the primary tool available (function calls and tags can be discussed in another post).

Tip: You can use Anthropic's prompt generator to create the initial system message and examples, then paste it into ChatGPT's custom GPT (Anthropic's UI is too bad to use directly): https://x.com/danielvlopes/status/1792978707201831155

Here's a simplified version of what I use for my Blog Posts custom GPT in ChatGPT:

# Identity 
You are a draft polisher assistant. Your job is to receive long-form poorly formatted drafts and convert them to a final version maintaining the tone, and vocabulary but using fewer words, fixing the grammar, adding more spacing, better structure, and making things clearer.

# Steps to take

To do this, first carefully read through the entire blog post to understand its structure and key points. Then, go through and rewrite the polished version of each section while keeping the same titles and technical insights.

Here are some tips:

- Remove unnecessarily complex language. Replace these with plain, straightforward terms. For example:
-- Instead of "leverage", use "use"  
-- Instead of "synergize", explain what you actually mean, like "work together"
-- Instead of "disintermediation", just explain the concept in simple terms
- Break up long, run-on sentences into shorter, clearer ones
- Use contractions like "it's" and "you'll" to make it sound more natural and conversational 
- Identify the technical parts of the text and don't remove these
- Keep the tone straightforward and professional, avoiding overly casual language but ensuring it doesn't become too formal.
- Keep the personal touch and original intent of the content.
- Format the condensed version of the post with clear section breaks and spacing for readability.

Here's a list of banned words to never ever use any of the following words (or similar words): "delve", "embarked", "journey", "thrilled", "aim", "tailored", "fledgling", "shrouded", "endeavor" , "quantifiable", "empirical", "whereabouts", "debut", "keen", "craft", "tailor", "elevate", "ignite", "empower", "unleash", "horizon", "harness", "diving into", "interconnect", "enhance".

As you write the new version, make sure to maintain the same voice, tone, vocabulary, and style as the original blog post. The goal is not to rewrite it completely, but to have a slight more polished version that is easily digestible format for people short on time.

# Example 1

Input: 
```
Original draft
```

Output: 
```
Well written version
```

# Example 2:

Input: 
```
Original draft
```

Output: 
```
Well written version
```

# Example 3:

Input: 
```
Original draft
```

Output: 
```
Well written version
```

Put in some effort; otherwise, you are to blame for the bad results. With early-stage technology, we can't have the luxury of expecting things to work perfectly right out of the box.

Idea selection

As I said before, I'm currently focused on small indie-hacker-scoped projects. What does that mean?

I'm sorting ideas by level of scope and potential size. For the first projects, I'm avoiding super ambitious things, they need large surface area and tend to have a lot of competition. In the middle, there are “during gold rush, sell shovels” type of ideas, like AI infrastructure tools for developers where the need is clear, but so many teams are rushing to build them.

I'm looking for small things or those with seemingly bad business models that I want to use myself. When picking a market, you either solve something for yourself or others. For now, I'm choosing things I want for myself to save time on initial customer research.

However, I still want something with clusters of people easily found online so marketing isn’t an uphill battle.

To recap: I need something not hard to market with a bad TAM, small in scope, and that I want for myself (likely small apps, wrappers, or add-ons for existing players).

Perfect recipe for disaster 😅 Why?

1. It gives me time to focus on adjacent things I'll reuse later, like a clean boilerplate with UI libraries, authentication, billing, and API clients.

2. In this first month, I'm basically starting easy to rest a bit while staying creative, studying, and even having a chance to build some baby MRR (but very unlikely).

I've narrowed my list down to 5 potential ideas for now, and I’m already working on a side project that I’ve had started in the past. These ideas will be for the next project.

Week recap: progress!

Boilerplate

Rails 🚂

I'm sticking with Rails - it's stable, battle-tested, and I know it well. Not interested in Next.js yet, maybe just for marketing later.

Going with the vanilla Rails stack for now, using Hotwire instead of React for the front end. Everything has a learning curve, and modern-day React is one I don't want to tackle now. Might use React later for more interactive ideas.

Jumpstart Pro 🚀

Didn't want to waste time setting up billing (even with Stripe), accounts, authentication, 2FA, or even little stuff like pagination. Went with Jumpstart Pro - it's a good starting point with high-quality code that's easy to customize, even if I'd prefer a smaller and with less dependencies. JSP also has a lot of new things that I wasn’t familiar with like Dev Containers, so fun learning experience.

TailwindCSS + TailwindUI (Pros & Cons) 🌬️

Switched from theming Bootstrap (like I did for Canopy) to TailwindUI. Not having CSS files makes it perfect for code generators like custom ChatGPT and V0, which are great at generating Tailwind markup. TailwindUI is also very versatile and covers a lot, so I get a productive boost from the (copy, paste, and customize).

However, using TailwindUI with vanilla Rails hasn't been smooth. I find myself coding basic things because TailwindUI has no vanilla JS option for behaviors like dropdowns, carousels, and off-canvas panels. Jumpstart's Stimulus helps with some of that. Another issue is that TailwindUI's markup was meant for React components, so you often need to abstract classes into Rails helpers for repetitive use.

This week: 2 days spent to cover TailwindUI basics, 1 day to learn Jumpstart, and the learning curve of TailwindCSS every day. But now feeling quite productive already.

Brainstorming with friends

I have a lot of people reaching out to network and brainstorm ideas. Last week I spent about & hours on this.

Canopy: GPT-4o

I work 8 hours a week at Canopy, doing 1-2 hours at a time. After OpenAI launched GPT-4o, I migrated our AI Assistant:

• It's much cheaper and faster (so fast that streaming isn’t necessary anymore)

• Overall better results: I tested using our rubric of 200 questions (not a formal eval system yet). After migration, the results were mostly better.

Two small issues:

1. Acronyms: It made up some acronyms, which I fixed with prompt engineering.

2. JSON: We use OpenAI's Assistant API with their RAG tool, so we can't use "JSON MODE". I ask for JSON via prompt engineering. It tried to add a wrapper around the markdown to avoid breaking the JSON, which we fixed by adjusting the code.

The speed and price drop let us split our 200+ line prompt into two prompts, running the query sequentially:

1. Ask for the answer, present it to the user

2. Ask for additional metadata async and refresh that part of the screen (WIP, shipping next week)

The cost drop opens up the opportunity to use GPT-4 level of IQ for everything, including B2C customers with low subscription prices.

Product 1: A Lot of Work Done

As I said before, I'm already working on my first product, which I've been building since early this year a few hours a week. It's a small thing, but a useful tool for myself that benefits from LLM support.

I spent 3 days (30 hours) migrating the code to Jumpstart Pro and 4o, and adding features for launch - shooting for next weekend. I also worked on the marketing landing page.

Still to do: Complete an “insights” feature. Add an LLM-powered overview screen. TailwindUI is helping a lot with the design. So, I’m estimating ~3 to 5 more days before able to onboard users.

I’ll need to setup new servers due to significant changes from Jumpstart Pro. Setting up servers again will take likely at most one day.

Goal: For this week or next, put the product up + add a marketing landing with a waitlist, and send it to the communities I’m a member of.

The bad things

1. Poor sleep hygiene: Worked too close to bed, messing up my sleep. Getting older I need to be a lot smarter about sleep hygiene these days or the price is real.

2. Not enough exercise: Only exercised 3 times: 2x 40min half-assed weight sessions, 45min hard VO2 max virtual cycling race on Zwift. Usually do 2x that, it's important for my mental health.

3. Missed meet-ups: Trying to go to more. Had two interesting ones this week, an Agent Guidance and a Gemini vs 4o Hackathon on Saturday, and skipped both.

Next week

I'm off for 3 days on a last-minute camping trip, so this week will be short. Working this weekend to account for it. Most of my time will be split between working on Product 1 and some LLM work for Canopy.