The standard definition: an ideal customer profile (ICP) is an account-level description of the companies you are best at winning, serving, retaining and expanding. It is a short list of firmographic and technographic criteria, plus explicit disqualifiers, scored and tiered in the CRM.

An ICP describes companies rather than people. A buyer persona is the person inside the account, TAM/SAM/SOM are market-size estimates in dollars, and an ABM target list is the named accounts you are working right now.

That definition is correct, and it produces some pretty useless docs. Sort your customers by industry and headcount and you get a profile that describes who they are while saying nothing about why they bought.

We wrote ours for the GrowthOS launch and now 2 months in revisited focused on the way round, starting from the job. Christensen: “The job, not the customer, is the fundamental unit of analysis.” Bob Moesta’s version changes what you do on a Tuesday: demand starts at a struggling moment and “it’s not an imagined customer or persona, it’s real buyers.”

So the first question is not what kind of company buys this. It is which organizations hit the struggling moment this product resolves, what progress they are trying to make, and what they are hiring today instead. Firmographics come back as filters on that answer, never as the explanation for it.

The other half of a real ICP is that it is enforceable: an SDR can disqualify an account against it, RevOps can score and route on it, and it gets reviewed monthly against closed-won data.

Revisiting our ICP

We wrote our ICP doc before GrowthOS launched in June 2026. It was a hypothesis with a template around it: which companies the platform would be structurally best at winning, extrapolated from work we had been doing for clients by hand. The first paid clients of the platform have since converted onto annual plans, which is the first evidence in the exercise that isn’t a guess, so we went back to the doc.

Before touching it, I created this study guide for myself, but I’m sharing it here since it might be a helpful resource.

A caveat on the numbers: they come from different years, samples and definitions, so several of them disagree. Use them as shape, not as truth.

Audio version

Here’s a computer generated audio version of this guide adapted for listening

What an ICP is, and what it isn’t (TAM, target lists, personas)

RevOps Co-op defines an ICP through four operational questions: who will buy, who will purchase smoothly and on time, who can be implemented and serviced, and who will renew, refer and expand.

Only the first is a sales question. An ICP built from “who signs” systematically overweights accounts that close and churn, which is why retention and expansion data belong in the derivation rather than in a CS dashboard.

The definitions worth having in front of you. HubSpot grounds it in data: an ICP is “based on company level firmographic and technographic data points (like employee size, technologies used, industry, and more).” First Round Review adds the temporal part: “a detailed description of the perfect customer (in the case of B2B, that means company)” and “a living definition that needs to be revisited frequently.” SaaStr raises the stakes: “Your ICP strategy is the basis for your entire company’s strategy.”

Four layers, four different questions:

• TAM/SAM/SOM works at the market aggregate. How large is the opportunity? The output is a dollar figure.

• The ICP works at the account level. Which types of companies fit best? The output is attribute criteria and a scoring model.

• An ABM target account list works at the named-account level. Which specific companies are we pursuing now? The output is a prioritized list of names.

• A buyer persona works at the individual level. Who inside those companies do we engage, and how? The output is a role, goal and pain profile.

The sequence runs: size the market, define ICP attributes, score the addressable universe, apply capacity and territory filters to get the named list, then map the buying committee with personas.

Clearbit draws the market-sizing line cleanly: “TAM, SAM, and SOM identify market potential and are represented by a dollar value. Your ICP takes that a step further and serves as a definition of the companies that are the best fit for your business.” Clay’s version: “every ICP account is in your TAM, but most TAM accounts are not in your ICP.”

On personas, HubSpot gives the operating split: “Think of your ICP as your pre-qualification filter and buyer personas as your personalization guide.” Forrester adds the complication that makes personas insufficient on their own: the same VP of HR can sit in several buying groups and play a different role in each. The five common roles are champion, influencer, decision-maker, user and ratifier.

An account can be a perfect ICP match and still sit outside the active target list on capacity or timing, which is why HubSpot’s ABM system tracks “target account” and “ICP tier” as two separate properties, Tier 1 being “a great fit” and Tier 3 “acceptable, but low priority.”

A fourth distinction is between ICP and IPP. Force Management draws the line “ideal customer profile does not necessarily equal ideal prospect profile.” A prestigious account can match the profile and still be a bad near-term prospect on stakeholder complexity or contract timing.

Jake Fuentes at Cascade gives the test for whether a candidate segment is real: “An ICP must be a single market segment: a group of people for whom the value of solving a problem is roughly the same, and who can be reached in roughly the same way.” Equal value and equal reachability, both necessary. OpenView arrives at the same test independently: a good segment “gets value in the same way and buys in the same way.”

Meka Asonye at First Round shows the difference between a vague segment and a usable one. Bad: “Series B technology companies.” Good: “companies who are experiencing X, who look like Y, who have previously tried these three solutions and need the product to do A, B, and C.”

Fit vs intent: why an ICP needs two scores

I’d install a fit-versus-intent model first. Clay defines fit as whether a company should buy from you, derived from stable attributes like size and stack. Intent is whether it’s looking right now, derived from recent signals like funding or hiring.

A high-fit, low-intent account is a nurture target. A high-intent, low-fit account is a distraction.

Score the two axes separately so you can treat them differently, then combine at routing. Fit is the eligibility gate; intent and timing set the tier. Bombora puts it from the data side: only a small portion of your ICP is in market at any time, and intent data exists “not to define your ICP, but to reveal which accounts are actively researching and should be prioritized now.”

The compelling event is why a static firmographic profile underperforms a signal-laden one. RevOps Masters reports its win/loss result: “The single most predictive factor in win/loss outcomes is whether the buyer had an identified event or deadline driving the purchase. Deals with a compelling event close at 3-4× the rate of those without one.”

Clearbit’s routing rule shows what combining the axes looks like in practice: “High-intent visitors should speak to sales right away if they have high fit. Low-intent visitors can be added to an ad retargeting campaign, email nurture sequence.” Their threshold example: an account that visited at least three relevant pages in a month, whose fit score qualified, got routed to the BDR team.

Which attributes belong in an ICP?

Four attribute families matter, and most bad ICPs stop at the first.

• Firmographic. The standard set: industry, company size, revenue range, employee count, geography and business model, plus ownership type. Bessemer’s bands: Commercial 1-500 employees, Mid-market 501-1,000, Enterprise 1,000+, with revenue bands of $0-1M, $1.1-10M and $10.1M+.

• Technographic. What they run: “which CRM they run, what marketing tools they rely on, where they host their infrastructure, and what sales platforms their reps use every day.” Two mid-market SaaS companies “might look identical on paper, but if one uses Salesforce with a modern tech stack and the other runs a legacy CRM with no API access, your approach should be completely different.” HG Insights goes further, measuring behind-the-firewall usage intensity and maturity, and separating Problem Aware, Solution Aware and Product Aware stages.

• Behavioral. Buying signals: content consumption, website engagement, event attendance, search intent. What separates companies that fit your ICP in theory from ones actively buying. In product-led motions this is where PQLs live, converting at 15-30% on signals like usage frequency, adoption of high-value features, usage growth week over week, and multi-product usage.

• Psychographic. How the organization decides. One list worth stealing covers “appetite for innovation, willingness to buy best-of-breed, attitude to vendors, decision-making style.” The pace difference matters too: “Some decisions require consensus across a large leadership team. Others sit with a single executive who acts quickly.” Lenny Rachitsky lists a company’s “way of working”, design-driven or operationally heavy, as an explicit field.

Prefer operational signals to generic firmographics wherever you can get them. First Round’s example is the cleanest illustration: instead of targeting companies with more than 500 employees, target “companies with more than 20 open, remote roles on LinkedIn. That’s a sharper ICP.” Same for pain: “It’s not enough to know the pain the prospect has, you want to deeply understand the impact of that pain.”

Bessemer’s five-factor needs framework, credited to Allyson Letteri, is the best short prompt list I’ve found for the situational half:

Pains: what pressing problems can your solution solve? Gains: what goals or desired outcomes drive your customer? Shifts: what company changes would make them open to new solutions? Blockers: what objections or misconceptions might hinder adoption? Motivators: what would urge them to move forward, ROI or testimonials or something else?

Shifts is the underused row. Common ones are “compliance mandates, a crisis or breach, new executives joining the team, or even when the team outgrows old systems.”

Product usage deserves its own note if you have a free tier. OpenView reports that Figma defined activation as “a user collaborates with other users in their first week” and modeled around 10 data points where “when two or more of these were triggered, there was a high likelihood for the account to upgrade.” A concrete multi-user trigger from the same playbook: when three or more users at the same company reach a high usage threshold, email the manager.

The jobs-to-be-done lens turns a tidy attribute list into a usable one. Tony Ulwick separates the job from the outcome, “a metric the customer uses to measure success”, which lets you cluster accounts by which desired outcomes are underserved and define segments that cut across industry and size bands. OpenView warns that conventional segmentation “can be severely deficient because it is an arbitrarily imposed view of the market,” and every criterion needs “a clear rationale as to why the particular criterion will impact or differentiate the needs or buying behavior of targets.”

Looker’s 2013-2015 ICP led with the job: “technical data teams, not end users or business analysts, who were starting to adopt cloud with large data sizes and complex analytical requirements AND the need to support a larger base of less technical end users.” Company size, 50-400 employees, appears only as a supporting filter and never as the explanation for demand.

How many attributes, and how narrow

Rachitsky collected the initial ICPs of more than a dozen B2B startups and found that “everyone landed on at least three attributes to describe their ICP. Some had more, but no one had fewer.” His instruction is to get “super-specific and super-narrow. Almost comically narrow.”

The receipts are the useful part:

• Gusto started with companies of five or fewer employees, in California, offering no benefits, salaried employees only, no other deductions, and willing to be paid eight days after running payroll.

• Gong picked three: selling in the US in English, selling via video conferencing, and selling software worth $1,000 to $100,000, “because beyond $100k, we assumed it was going to be a different sales cycle, and less than $1,000, it would be too transactional.” That left roughly 5,000 companies worldwide.

• Snyk went depth-first: “a developer building with Node.js who was very security-conscious.”

• Canva found theirs six months in, watching who got excited: social media managers and bloggers, “especially freelancers, who were building their own social media management business.”

At the other end, keep the filter to 4-7 attributes. More than 8-10 filtering dimensions and it is over-specified, fitting noise, and no longer usable by an SDR in real time.

Five frameworks for choosing a segment

Each framework contributes a different narrowing mechanism. You don’t need all five, but you should know which one you’re using.

In one line each: Moore picks the first segment by use-case urgency and reference community. Blank validates that a specific org type and buyer will actually purchase. Lean Analytics replaces aggregate counts with cohort metrics. MEDDPICC tests reality inside an account and aggregates into buyability patterns. Top-10 reverse engineering derives the profile from realized outcomes.

Moore. The transition that matters is early adopters to pragmatists who “want to see it proven out first, specifically in use cases that they themselves have and with customers they know and can reference.” A market is defined partly by reference behavior customers must “reference each other when making a buying decision.” Choose the first segment on problem severity rather than size,: “The size of the first pin is not the issue, but the economic value of the problem it fixes is. The more serious the problem, the faster the target niche will pull you out of the chasm.” The ICP has to encode the whole product, “the complete set of products and services needed to fulfill the compelling reason to buy,” which doubles as a test of whether you can actually serve the segment. Expansion then follows the bowling alley: target a connected segment that “by virtue of its other connections, creates an entry point into a larger segment.”

Blank. Customer Development runs discovery, validation, creation, company building. The earlyvangelist five criteria are the pre-ICP filter: “They have a problem. They understand they have a problem. They are actively searching for a solution and have a timetable for finding it. The problem is painful enough that they have cobbled together an interim solution. They have, or can quickly acquire, dollars to purchase the product.” He also advises going after companies that “aren’t the market leaders in their industries, but are fighting hard to get there,” then finding the internal evangelist who wants the competitive advantage. The validated sales model has to answer who influences, who recommends, who decides, who pays, where the budget sits, what acquisition costs, and how long a sale takes.

Lean Analytics. Cohort discipline instead of aggregate counts since “you can compare cohorts against one another to see if, on the whole, key metrics are getting better over time.” The OMTM is stage-dependent and temporary, and in interviews you “look for a subset of scores that spike; that’s your early-adopter customer segment.” Three cohorts, three questions: acquisition and win/loss (”can we get them?”), retention and GRR (”do they stay?”), expansion and NRR (”do they grow?”). The method needs an overfit warning: “if accounts that match your candidate ICP also churned at the highest rate, you have over-fit on a non-causal pattern and need to add an exclusion criterion.”

MEDDIC/MEDDPICC. Metrics, Economic buyer, Decision criteria, Decision process, Paper process, Identify pain, Champion, Competition, originally developed in 1996 by Dick Dunkel at PTC, with Paper Process and Competition added later. MEDDPICC contributes the feedback loop. Aggregate outcomes across won, lost, stalled and no-decision deals, tagged by account attributes, and you learn which characteristics predict buyability, cycle time and commercial success rather than theoretical fit.

Top-10 reverse engineering. Derive the profile from realized outcomes. SaaStr’s questions for your best deals: “How easy is it to close these top deals? How simple was it to set your customer up with onboarding and implementation? How sticky is your product?” A working version analyzes “10 to 20 best existing customers across revenue contribution, retention rate, sales cycle length, and product fit.” Allston Labs is the most rigorous: capture three layers per customer, “firmographic (who they are), behavioral (how they bought), and deal-shape (the buying committee that signed)... including the nulls, the distribution of nulls is itself a signal.” Weight the expansion cohort 3-5× because repeat purchase reveals organizational fit as well as product fit, treat closed-lost as “the empirical control group,” and note their headline claim: “the single highest-confidence ICP indicator is per-customer ACV growth at 12 months.”

How do you build an ICP from data?

Six analyses, in the order to run them.

1. Closed-won and closed-lost

Build a joinable dataset across six layers: opportunity, account, contact, source and campaign, product and use case, and post-sale retention. Separate new-logo, expansion and renewal cohorts before analyzing. Use a two-level loss taxonomy: primary as competitive, no decision, price and budget, or product gap; secondary as feature depth, integration ecosystem, implementation timeline.

Pull the last 12-24 months. No-decision outcomes are 25-40% of closed-lost deals on their own and should never be merged into competitor or price losses.

How much data you need before believing the pattern: 5-7 closed-won deals support a provisional hypothesis and nothing more, 30 closed-won is a stronger first-pass minimum, and 50 combined closed deals supports general win/loss pattern analysis.

Two behavioral signals worth pulling in the same query: hiring activity is associated with a 2.4× conversion lift, and funding events indicate a 3-6 month timing window. And triangulate rather than trusting one source: RevOps Masters recommends CRM loss reasons, rep debrief surveys and buyer interviews together, because rep-entered reasons measure perception.

A defensible scoring starting point, if you want one before you have enough data for a model: a 0-100 scorecard weighted firmographic fit 30, technographic fit 25, behavioral signals 25, deal potential 20.

2. Win rate and cycle length by segment

Never report blended. The example that makes it obvious: “A 24% aggregate win rate breaking down to 41% for SMB inbound and 9% for enterprise outbound tells you everything.”

The formulas, so the segmentation happens before the math: win rate by count is won deals divided by won plus comparable closed-lost, and sales velocity is opportunities × deal value × win rate ÷ cycle length. Segment by market size first, then calculate.

Across 4.2M opportunities at 530 companies and $54B of revenue, Ebsta and Pavilion found top performers had 42% shorter cycles, 76% higher ACV and 43% higher win rates than average. Early decision-maker involvement in the first two stages lifted win rates 55%, high-intent accounts closed at 3.4× velocity, and top performers were 24% more likely to disqualify non-ICP deals early. For orientation, average B2B win rates run 15-25% with mid-market cycles of 30-90 days, and lost deals take 2.0× longer than won ones.

One data-quality caveat before you trust any of your own numbers: Clari Labs, across 10M opportunities at 121 global enterprises, found 98% of companies fail to track closed-lost reasons consistently.

3. Unit economics by segment

Read gross-margin-adjusted CAC payback together with net dollar retention rather than alone, since a segment with short payback but poor retention is misleading. The formula: sales and marketing spend to acquire new customers ÷ (contracted new ARR × gross margin) × 12.

Bessemer grades it good at 12-18 months, better at 6-12, best at 0-6, with segment targets in favorable conditions of SMB under 12, mid-market under 18, enterprise under 24.

OpenView measured it:

by target customer, as good (50th percentile) against great (80th): under 20 employees, 9 months against 2; SMB at 20-100 employees, 7 against 4; midmarket at 101-1,000, 14 against 7; enterprise above 1,000, 14 against 9.

KeyBanc’s 2024 survey puts fully-loaded payback at 25 months in 2022, 21 in 2023 and 20 estimated for 2024. These figures disagree because the years, samples, ARR scales and definitions differ. Shape, not truth.

4. Retention cohorts

GRR is the stickiness check, NRR the expansion check, and a strong NRR can hide widespread churn behind a few large expansions. The formulas: NRR includes expansion and reactivation and can exceed 100%; GRR excludes expansion and caps at 100%.

Benchmarks worth arguing with: Bessemer’s good/better/best is NRR 100/110/120%+ with logo retention above 85/90/95%; best-in-class B2B NRR sits at 110-125% with median monthly churn of 3.7% for $1-3M ARR companies; ICONIQ’s 2025 report has software NDR settling around 110-120%; and OpenView’s 2023 data shows PLG companies at 105% NDR against 98% for non-PLG.

The number that justifies the whole exercise, from OpenView: “annual retention rates can vary from 50% to 90% across different customer types for the same product.”

In Clearbit’s worked example, ~86% of long-term revenue came from ~18% of leads entering the funnel, and they narrowed the ICP accordingly.

5. Usage and time-to-value

Median SaaS activation is 30%, average 36%, and activated users should retain at 2× non-activated. For B2B, activation typically takes two to three weeks rather than minutes. Similarweb scores PQLs on two factors, ICP attributes (company size, role, use intent) and usage behavior (frequency and breadth), combining them to set engagement level and which product to sell.

Strong-fit usage looks like fast time to value, high activation against matched non-ICP cohorts, repeated use of the core workflow, adoption of the differentiating feature, multiple active users or departments, and usage rising over time.

Weak fit looks like signup without ever reaching the aha, shallow single-feature or one-off use, low account breadth, frequency declining after the trial, a high support and onboarding burden, and no expansion path.

6. Interviews

Quantitative data alone misses the mechanism, and there is a number for how badly. Clozd reports that buyer and seller reasons for closed-lost deals “only align 15% of the time, meaning 85% of CRM closed-lost data is potentially inaccurate,” and that roughly 70% of buyers name a different primary competitor than the CRM has. Surveys alone don’t fix it either: response rates in some B2B sectors run under 2%.

The thresholds worth knowing: five to eight conversations for a focused discovery sprint, ten similar deals as the minimum cohort to see patterns, and twenty within a segment before making a major strategic change.

Jason Sippey, formerly VP Product at Twitter, describes the progression: “After the first 10, you start to see patterns. After 20, you really understand segmentation of the market. After 30, you have a really good understanding of what you actually need to go build.” His four questions: do you have this problem, how are you solving it today, how much are you spending to solve it, how does it impact your business.

For loss interviews specifically, GTM Operations’ five questions are better than most scripts: what triggered the search, which vendors made the shortlist and what were the first impressions, rank the top three factors in the decision, was there a moment your opinion of us shifted, and what would have had to be different for the outcome to change.

Two discipline rules. Buyers should speak 90% of the time. And the payoff for keeping at it: companies running win-loss for two or more years report an 84% increase in win rate.

The same interviews give four qualitative signs that you are closing in on the right segment: a significant jump in conversion rate, a significant jump in enthusiasm, a much stronger desire to act now, and the nod.

Sequence: quantitative analysis to find anomalous or attractive cohorts, qualitative interviews to find the mechanism and the language, then a second quantitative test for prevalence and predictive power.

Working with a small sample

You will almost always be here first. The workflow with roughly 12 accounts: pull the top 3 by revenue and top 3 by NPS or NRR, find the shared attributes, and treat the result as a hypothesis you revise quarterly.

Then borrow four rules from Allston Labs: use closed-lost as the control group, since attributes that differ between won and lost carry the decision weight; compute a win/loss ratio per attribute, where something appearing in 70% of won and 30% of lost is high signal; weight expansion cohorts 3-5×; and run three to five structured interviews with your most engaged customers.

What should an ICP document include?

Assembled from James Doman-Pipe’s GTM Playbook template, Full Funnel’s ABM six pillars and HubSpot’s partner ICP worksheet.

1. Header. Name, owner, last updated, change log. HubSpot’s is the only one of the three with a change log, and skipping it is how a definition goes stale without anyone noticing.

2. Firmographics. Industry, employee count, revenue band, geography, business model, funding stage and growth rate.

3. Technographics. Current stack, competitive and complementary solutions in use, IT spend and digital maturity.

4. Situational fit. Pain signal, buying trigger, current solution, budget parameters. Doman-Pipe’s bar for a pain signal is observable rather than adjectival,: “Not ‘they have messy data’ but ‘their team is manually reconciling data from three tools every Monday morning and it is taking four hours.’ Specific and observable.”

5. Buying committee. Economic buyer and what they care about, champion and the pain they feel, influencers, blockers and insiders, and day-to-day users versus buyers.

6. Why we win here. Two or three outcomes delivered, plus the structural advantage in this segment. If you can’t name the advantage, this is a segment you can serve rather than one to commit to.

7. Disqualifiers. Their own section, not the inverse of the positive criteria.

8. CRM fields. Fit score 0-100 and ICP tier as properties on the company object, how we find them, owner, last review date.

Published templates worth copying

• a16z’s nine fields: company size or revenue range, type of business, geography, industries served, job titles, a definable solvable problem, company-specific attributes, specific technologies used, unique buyer behaviors. With five diagnostic questions: which customers get the most out of the product, what traits do the best share, what objections recur in losses, who is easiest to upsell, and what do competitors’ customers have in common. Their worked example: “Large global retailers with onsite developer teams that use GitHub; have specific requirements for PII and customer data residence drive vendor decisions; and their cloud usage means they don’t have on premise requirements.”

• Lenny’s nine fields: company size, job title, pain point being solved, company’s unique way of working, specific tech used, type of business, price point, geography, and a unique place the user spends time.

• Bessemer’s Six Ps worksheet: Persona, Problem, Proposition, Product, Positioning, Promotion, with a bar for the persona section I like a lot: if all the criteria are met, the account should buy 80% of the time. Their worked example rejected universities as an ICP because “the higher education market is incredibly fragmented, insular, has restrictive budgets, and quite political,” and moved to residency teaching hospitals, clinics and paramedic schools.

• Productboard’s seven dimensions: company size, product stage, digital-first or digital transformation, single or multiple products, size of the product organization, care for the customer, B2B or B2C. Which resolved to “an early stage, digital-first startup with strong customer-centric, product-led culture, with one product team, building a single (ideally) B2B product.”

• HubSpot’s own 2024 NAM/EMEA ICP, which is rare in publishing its own numbers: 150+ employees, $100M-$500M revenue, software, IT and financial services, $4.6K average monthly deal size, 25% conversion rate, 75-90 day deal length.

• Adam Schoenfeld’s paragraph format, for teams that won’t read a table: “We are best for B2B SaaS companies with 75-5,000 employees in the US and Canada. Our ideal customers are growing revenue with a sales-led motion, have 10+ AEs assigned to account-based territories, have sophisticated revenue operations...”

• GTM Labs’ worked example is the one I’d hold up as the standard for specificity: 100-500 engineer companies on Kubernetes spending $5K+/month on Datadog, where a Senior SRE is the technical champion and the VP Engineering owns the budget, triggered by a Datadog renewal within 60 days.

The disqualifier section

A negative ICP is a distinct, named section. João Coelho treats them as separate architecture: “Disqualification criteria are not the inverse of ICP. They’re a separate architectural decision.”

Three exclusion types: hard (never target, such as a size floor or a country you cannot legally support), soft (target only with a different motion, such as enterprise logos needing field sales rather than cold email), and conditional (target only if a trigger overrides the default, such as a smaller company that just raised).

Concrete dimensions people actually write down: pre-revenue companies with no paid motion, or sub-200 employees when the deal size is too small; agencies reselling without their own list, or project-based engagements under $25K; a champion with no budget authority and no path to a decision-maker; missing core integrations; and a company eight months into a three-year contract with a direct competitor.

Then make it mechanical. A multi-signal routing rule: three or more negative signals auto-disqualifies, two routes to manual review, one proceeds but flags in the CRM. And the enforcement point that matters most: “Do not rely on rep discretion. Use CRM views, enrichment rules, and sequence entry criteria. If a segment is truly a negative ICP, the system should block it before it reaches a sender account.”

The CRO at Consensus built the positive version of this out of win-loss data and called it a UCP, an un-ideal customer profile: “We now know exactly what a bad prospect looks like,” which they tied to a more streamlined sales process and lower churn.

Who owns it, and where it lives

Two artifacts, not one. The narrative version lives in a shared doc: Full Funnel publishes a Google Sheet, HubSpot a PDF worksheet. The enforced version lives in the CRM as an ICP fit score (number, 0-100) and an ICP tier (dropdown: Tier 1, 2, 3, Not a fit) on the company object.

Ownership goes to RevOps where that function exists, “or if you don’t have RevOps yet, whoever manages the CRM and the go-to-market cadence.” Earlier than that it belongs to the founders because “when you’re super early, everything should still be sales-led. You’re still figuring out your ICP and personas, and that requires talking to customers.” When conversion is inconsistent rather than absent, product marketing is the better owner. HubSpot’s cadence: “review it every quarter using fresh CRM data, closed-won deal patterns, and updated customer feedback.”

How do you score and tier accounts?

Four scoring models require progressively more data.

• Weighted point-based, the simplest thing that works. 500+ employees +50, Director title +10, industry, country and technology 1-5 each, negative values for non-ICP attributes. Clearbit describes the progression from basic firmographic filtering to weighted points to predictive models, which is the order to walk it.

• Weighted rubric. A two-layer model: industry 20, employee count 20, revenue band 15, tech-stack adjacency 15, geography 10, funding and growth 10, hiring 10, each scored 100/60/30/0 by band. Combined as Account Score =(0.4 × Fit) + (0.6 × Intent), with intent recency multipliers of 1.0× for 0-30 days, 0.5× for 30-90, 0.1× beyond. A published variant weights firmographics 40%, technographics 25%, intent 20%, behavior 15%, with 70+ qualified, 50-69 worth working, under 50 disqualified. Another is industry 25%, size 20%, tech stack 15%, growth signals 15%, org maturity 10%, geography 10%, negative filters 5%.

• Decision-tree look-alike. MadKudu’s model uses firmographic, demographic and technographic data only for fit, with behavior in a separate Likelihood to Buy model recomputed several times daily and combined as Lead Grade = 2× Fit + 1× LTB. Bands: Very Good 85-100, Good 70-84, Medium 50-69, Low 0-49. Validation targets: recall above 70%, precision as a 10× conversion ratio between top and bottom scores, rejection rate under 5% for top scores.

• Customer-specific predictive. 6sense trains a model per customer on firmographics, technographics and custom fields, and the model itself identifies which characteristics matter. Bands Strong 81-100, Moderate 50-80, Weak 0-49, updated daily.

Two rules apply across all four models. Hard disqualifiers stay binary: never let a high firmographic score buy its way past an exclusion. And if you have two or three segments with meaningfully different win criteria, build separate models, because blending obscures the signal.

Tiering

ICP fit is the eligibility gate; intent, timing and signal strength determine tier placement. Composite formulas exist if you want one: (ICP Fit × 0.30) + (Intent × 0.25) + (Engagement × 0.25) + (Revenue Potential × 0.20), or fit 30%, revenue potential 25%, intent 20%, engagement 15%, strategic value 10%.

What matters more is tying each tier to a motion and a volume rather than to enthusiasm:

Tier 1 is a perfect fit across all dimensions, $100K+ ACV in the worked example, 10-25 accounts per rep, custom outbound with exec engagement and ABM ads, five to eight touches a week, fully personalized.

Tier 2 is a strong fit with one or two minor mismatches, $25K-$100K, 25-50 accounts per rep, programmatic ABM with industry-segmented messaging, two or three touches a week, semi-personalized.

Tier 3 is a partial fit that clears the minimum, under $25K, 100+ accounts per rep, inbound and paid social and self-serve, one automated touch, templated.

For calibration, HubSpot runs Tier 1 at 20-50 accounts with “deep research and one-to-one customized outreach” and Tier 2 at roughly 200 with industry and persona personalization. ACV maps to load: 20-50 accounts per rep in field sales at $50K-$500K ACV, 50-150 in a two-stage motion at $10K-$50K. Mutiny shows what real separation looks like: Tier A converted lead-to-opportunity at 30%, Tier B at 8%, and Tiers C and D were “effectively rounding errors.” Oliver Jay started with 100 enterprise accounts at Asana and concluded “that’s too many. It should have been just 50.”

Tiers should move. Mosaic narrowed 100,000 accounts to 8,000 managed by nine BDRs, with accounts crossing tiers annually as funding, size, geography and stack change. One team re-tiered its entire customer base in three days on four signals (growth potential, AI maturity, engagement, account health), landing on 8 Tier A accounts and 21 Tier B.

Trigger events

Six categories, across 6sense, Bombora, ZoomInfo, Demandbase, Common Room and UserGems: capital and funding; leadership changes, where UserGems reports past champions “convert about 3x higher than normal leads”; hiring and job postings, which reveal strategic priorities; technology adoption or replacement, where HG Insights’ technographic flags produce up to a 5x conversion rate to opportunity; expansion news; and layoffs or contraction, which can indicate consolidation needs.

First-party intent has its own ladder, from Clearbit: high intent is pricing, SKU, trial and demo pages; medium is customer stories, solutions, partners and integrations; low is the homepage, about page and blog.

Measuring the ICP itself

Not just the accounts. ICP pipeline contribution as a share of opportunities by count and by dollars, win rate and deal velocity by tier, and renewal rate by tier.

Bessemer stages the reporting: “At $1M ARR it is common to look at total pipeline, but at $25-50M this pipeline should be segmented by ICP targets” and split new-logo from expansion. Coverage benchmarks to hold it against: 3-5× the ARR goal unweighted, 2×+ weighted in-period, and $8-10 of pipeline per $1 of marketing spend.

How should an ICP change by stage, and how often should you revisit it?

Pre-PMF: uncomfortably narrow. Paul Graham’s version is to satisfy “all the needs of a subset of potential users” rather than a subset of the needs of all of them. YC: “recruiting 10 customers who have a burning problem is much better than 1000 customers who have a passing annoyance.” Bessemer says to make it narrow “uncomfortably narrow, so narrow it almost feels too small,” to focus on one ICP at a time even when the product has broad applications, and to avoid targeting enterprises first. First Round’s Emery Rosansky sets the epistemics: “in the early days, you’re flying nearly blind with a small amount of data, so this initial ICP is not much more than an educated hypothesis.”

The obstacle at this stage is usually not ignorance. Clay’s Kareem Amin names it: “Often it’s not that you don’t know who the customer is, it’s that you’re not picking, you haven’t committed to one hypothesis over the other.” When Clay did narrow, it enforced the choice by cutting irrelevant features, removing non-tailored marketing language, updating internal docs and re-educating the team. Cadence here is event-driven, not calendar-driven.

Two findings from Rachitsky’s founder interviews pull in opposite directions. Most founders got their first ICP wrong, so the exercise is iteration rather than insight: Persona’s Rick Song ran it “17 times in the early days,” and Databricks had none at all, working with “a hospital that was using this stuff... folks that were using us to determine earthquake magnitudes using Twitter.”

Skipping the exercise can slow product-market fit. Mathilde Collin at Front: “We did not think about ICP. I wish we did earlier on. It’s one of my biggest mistakes.” Barry McCardel at Hex was forced into it by launch: “once we did our public launch, we started getting a flood of different people coming in. Filtering through the leads spurred me into putting a much tighter definition around qualification for deals.”

Series A/B: evidence-driven narrowing. a16z frames the stage “rapidly experimenting to refine your ideal customer profile and build repeatable sales motions.” Bessemer’s ARR ladder is the most specific guidance available: at $1-10M ARR, “aim to fulfill the product vision for your initial ideal customer profile,” sequencing product-market fit, then strong net retention in the ICP base, then broadening; at $10-25M, “it becomes important to start segmenting leads to the salespeople and sales teams who are best equipped to handle them”; at $25-50M, “crystallizing your ideal customer profile” becomes one of the most important tasks, and you finally have enough deals to find the pattern.

OpenView’s analytical method for this stage: “Pull all of your opportunities from the past year and clean the data. Look for the expected LTV per opportunity versus the CAC across different characteristics, such as industry, company size, buyer persona and use case.” With the caution that firmographics alone “does not drive any particularly important insights.”

Jason Lemkin puts numbers on when to split the team: begin segmenting “as early as 3 reps,” formalize three or four categories “after maybe $8m-$10m ARR.” Stripe added one segmentation dimension per year starting with size, and Jeanne DeWitt Grosser describes the failure mode: “You know you have a segmentation problem when the same account executive is talking to a million-dollar company in the morning and a billion-dollar company in the afternoon. Those sales processes have nothing to do with one another.” Cadence: a quarterly playbook review, and Mandy Cole’s instruction is to check it against reality: “Go through and see if it’s what you’re doing in the field, and if it’s not, update it to see the impact on pipeline.”

Enterprise: a new ICP, not an extension. Joe Morrissey at a16z warns against treating interest as fit: “the single biggest mistake I’ve seen companies make when moving upmarket is mistaking initial enterprise interest for product-market fit. Just because you have product-market fit for startups, SMBs, or mid-market customers does not mean you’ll automatically find product-market fit in the enterprise space.” Segment’s enterprise ICP was “larger enterprises that were B2C, multi-product, multi-brand, multi-subsidiary, and most importantly, data-literate,” narrowed further to those with “forward-thinking data champion buyers who were either decision-makers or could partner with a CMO to shape the decision criteria,” and it required outbound pipeline generation with 9-12 month cycles.

Budget for the new segment differently. Lemkin: “If you force your CAC in a new segment to hit the same ROI as your overall, blended CAC goals, you’ll never leave your core safe ICP,” so 80% of new customers should hit a sustainable CAC while the 20% in new segments “just try to barely break even,” with 24 months to converge.

At this size the data team becomes the reality check. Bessemer: “The data team will come in and identify the users who have high product engagement, retention, and reach, and in many cases, the attributes of this cohort of users will differ, at least somewhat, from an original conception of the ideal customer.” And a16z’s point about who holds the pieces: “You can’t define your ICP in a silo. RevOps has data. Product sees patterns. Sales hears objections. Marketing picks up signal. Success sees who churns and who expands.”

Review versus revision. The cadence argument resolves once you separate the two words. Review is checking evidence: monthly or quarterly against live closed-won data, which is what Anis Bennaceur at Attention.com means by “the teams compounding fastest are refreshing it off live closed-won data on a monthly or quarterly loop.” Revision is changing the operating definition, and it should be rare and trigger-based: a new product capability that changes who you can serve, retention, win rates, service costs or willingness to pay differing by segment, a move upmarket, or behavioral data diverging from the original conception.

The apparent disagreement in the sources resolves the same way. Daria Dovzhikova supports full rebuilds after major launches or funding rounds; Erik Miller warns “Don’t change the ICP every quarter; that creates whiplash for sales and marketing.” Both are right, about different words.

How the ICP shows up downstream: positioning, outbound, roadmap, board

A definition nothing enforces isn’t a definition.

Positioning. April Dunford defines positioning “how your product is a leader at delivering something that a well-defined set of customers cares a lot about,” and the order of her five components is the argument for where the ICP sits: competitive alternatives (”what would customers do if our solution didn’t exist?”), differentiated capabilities, value, then target customer segmentation, then market category. The ICP is derived from unique value rather than assumed before it, because “your best-fit target customers are customers that really care a lot about your unique value.” Starting from market category is “the tail wagging the dog.”

The payoff, in her words: “Customers that are well suited for your offering will easily understand your value, will purchase faster, and are much less likely to ask for discounts.” The canonical case is repositioning a general enterprise CRM, which invited comparison with Siebel, as a CRM for investment banks where modeling interpersonal relationships was mission-critical: “from under $2M to close to $80M” in 18 months.

Demand generation. Chris Walker’s paid-social targeting formula is just “(1) Job Title + (2) Company firmographics (named accounts, company size, industry),” with the classic errors being no job-title targeting, no firmographic layer, and LinkedIn audience expansion that makes you “pay high CPMs to people you don’t want to reach.” Refine Labs allocates 60-70% of paid media to demand creation and about 30% to capture.

Budget for the measurement gap while you’re at it. Across 620 declared-intent conversions and $21.5M in closed-won ARR, Refine Labs found a 90% gap between software attribution and first-party self-reported data: podcast was credited with 53% of revenue self-reported and 0% by software.

Three worked outcomes. Clari moved to target account lists, persona alignment and zero-click content instead of lead-gen forms: ad spend down 38%, cost per SQO down 36%, acquisition cost down 67%, win rates up 64%. Loxo built priority account lists with buyer-count exclusions and ICP-focused content: acquisition cost down 23%, ARR up 45% quarter over quarter. Zappi analyzed region, industry and persona out of the CRM and targeted by job function: 3× average deal size and 7× qualified pipeline against spend.

Outbound. Clay’s method is the ICP definition and the list build in one motion: “Pull the deals that closed fast, stayed, and expanded, and ask what they had in common the quarter before they bought. The patterns that repeat are your ICP. Write the definition down as a list of filters you can source against later, split into firmographic (size, industry, geography), technographic (the tools they run), and signal-based (funding, hiring, expansion).”

Apply disqualifiers before list export since “disqualifiers applied after export waste enrichment credits and create compliance risk.” The workflow most teams end up with: build criteria in Sales Navigator, export, run through Apollo or ZoomInfo for contact data, layer Crunchbase for funding, overlay BuiltWith for technographics, process through Clay for dedup and scoring. With one caveat worth pinning up: “Company headcount is the least reliable filter in Sales Navigator. Many companies mischaracterize their size or list no headcount at all.”

Content. The operating chain is ICP and account fit, then priority personas and buying situations, then topic pillars, then stage-appropriate formats, then channels. Every campaign, product and content strategy should align to at least one persona, built on “motivations, goals, pain points, and language, not only demographics.” Exit Five’s practical step is the one teams skip: “Create a One-Pager: Summarize your ICP in one clear document that your entire team can use.” For volume calibration, 47% of buyers view three to five pieces of content before talking to a rep.

Roadmap. The ICP “should be a formal input into quarterly roadmap reviews, not just a Marketing document.” A segment-weighted score makes the tradeoff explicit: revenue × strategic fit × growth signal × usage depth. Worked: a $30K mid-market account that is a core ICP match, expanding, and a power user scores 108; a $100K enterprise outlier that is contracting and lightly used scores 5, making “the mid-market account’s feedback 21x more valuable for prioritization purposes.” The counterweight keeps it honest: over-indexing on power users “harms the product experience for the rest of the user base, and steals product effort away from features that could go towards acquiring new users.”

Board reporting. a16z flags Net New Weighted Pipeline as the leading indicator of market pull. Pair each metric with the decision it drove, since 70% of a board meeting should focus on the future and the issues at hand. First Round’s Vanta example is the model: a 6% win rate on one product-buyer combination where “generally, a win rate below 30% means salespeople are wasting time and you need an unsustainable amount of pipeline. In this case, the solution wasn’t to fix sales, but to immediately stop selling this product to this buyer.” And SaaStr’s observed NRR gradient by deal size makes the segment story concrete: single-seat deals under $99/month at roughly 3% monthly churn, $99-$999 at about 100% NRR, $10K-$100K+ at about 120%.

Why do ICPs fail?

Fullcast and Pavilion’s 2026 GTM benchmark found ICP misalignment cuts win rates by up to 75%, and that 63% of 118 CROs had little or no confidence in their own ICP. Nobody in that survey is missing a document. They just don’t believe theirs.

Nine failure modes, in the order I’d expect to hit them rather than the order they’re usually listed.

1. Written but not operationalized. The pattern: “ICP alignment is not a strategy doc... You defined your ICP in a slide deck three quarters ago. But you haven’t updated routing rules since.” Owner.com’s CRO Kyle Norton found “either bad or no targeting on prospects,” poor-fit closes and mis-set expectations, with churn a “massive drag on efficiency” in his first 90 days; his fix was to pause the Growth plan, say no to a lot of customers, and build a lead quality system. Remedy: audit the top 50 closed-won deals against current scoring and routing rules. Figma’s version was to segment sales at the 1,000-employee line and pipe product usage into Salesforce via reverse ETL.

2. Static. One RevOps lead describes a client that “re-wrote their ICP properly last year, negative criteria and all, then left the old scoring weights and routing rules untouched for two quarters. The doc said one thing, the system kept qualifying the drifted accounts, and nobody reconciled the two until win rate forced it,” calling drift a “silent revenue killer manifesting as declining win rates, lengthening sales cycles, and churn concentration.” Lemkin’s line: “an annual ICP exercise is a 2021 habit.” Remedy: treat it as a living hypothesis with a monthly closed-won review. One founder who did that cut outbound from 2,000 to 400 emails a month while win rate rose from ~3% to 11%.

3. Aspirational. The written ICP describes the customer the company wants; closed-won describes the one it has. The cost: “The gap between those two is where marketing spend goes to die,” with an SDR team hitting an 11% win rate and a 178-day cycle against the written ICP. Sometimes the gap is a level, not a segment: First Round documents a company that “thought they were selling to CISOs at early- and growth-stage orgs, but realized through user research discussions that the people that acutely felt the pain were 2-3 levels below a CISO in an Enterprise org.” Remedy: start from 18-24 months of closed-won, and note Rachitsky’s finding that data from outbound sales beats leads from investors and friends as a signal.

4. Too broad. Cascade’s Jake Fuentes defined “nontechnical business analysts using Excel to crunch big data sets,” later calling it “much too broad” and describing the result as ICP fray: a scooter company managing location data, an HR team and a retailer under one profile. “To know what problem you’re solving, you need to know who you’re solving it for.” Remedy: replace generic firmographics with operational signals, and start from about three distinctive attributes.

5. Demographics instead of behavior and situation. A profile of static firmographics with nothing distinguishing accounts likely to buy now. Remedy: convert filters into real-time signals, and mine what you already own. Lemkin: prospect call transcripts are “the highest-signal first-party data your company will ever own.”

6. No negative ICP. No-decision deals “can account for up to 40% of a pipeline,” which is exactly what disqualifiers are for. Remedy: Mandy Cole’s Green/Yellow/Red: green is proactive pursuit, yellow is inbound and referral only, red is no-go.

7. No discipline at close. Quota pressure closes poor-fit deals. Katrina Wong separates an ICP from inbound demand: an ICP is a choice about “who you want to sell to versus who just wants to buy from you.” Clay’s co-founder Varun Anand gives the honest version of what discipline costs: “We had few true customers because almost none of the existing ones fit our new ICP,” and after narrowing to outbound sales, “nearly all of our original customers churned.”

8. Confusing ICP with persona. Remedy: keep the split clean: “ICP answers ‘what kind of company should we go after.’ Persona answers ‘who inside that company makes the call.’” Qualify the account first, then map the committee.

9. Trusting CRM loss reasons. 15% buyer-seller alignment, 70% competitor mismatch, 98% inconsistent tracking, all cited above. Remedy: 20 interviews within a segment before concluding anything.

The single best test of whether yours is real: an SDR can identify or disqualify an account quickly, marketing can target it, RevOps can score and route it, and sales can explain why it should be pursued now. If any of the four can’t, it’s a point of view rather than an ICP.

Where the practice is heading

The ICP-as-PDF is explicitly retired in current practice. iCustomer, July 2025: “The traditional definition of an Ideal Customer Profile as a static document describing your perfect customer is obsolete. In 2025, an ICP isn’t a document, it’s a dynamic, data-driven automated system that evolves in real-time based on live signals and market intelligence.”

Tooling now derives the profile directly. ZoomInfo’s AI-Generated ICP takes a CRM report of won and lost deals and learns which attributes indicate best targets, refreshing as conditions change. HG Insights’ Market Analyzer Copilot builds ICPs from win/loss, firmographic and technographic data correlated with highest-CLV customers.

The capability gap is the more interesting number. Enlyft found 81% of respondents likely to update their ICP in 2024, yet nearly 70% lacked a strong grasp of account scoring and 43% had invested in scoring without understanding its efficacy. The differentiator is not access to a model. It’s knowing what a good one looks like.

Further reading

• Lenny Rachitsky, How to identify your ideal customer profile. The operator counterpart to this page, and the best collection of initial ICPs anywhere: a nine-item attribute picker, a dozen founder accounts, and the three-attribute floor. It paywalls at the template and the comparison chart.

• a16z, a framework to define and refine your ICP, for the nine-field template and the revision triggers.

• First Round Review, the most common go-to-market questions from founders, which is where the hiring-signal example and the “educated hypothesis” framing come from.

• April Dunford, a quickstart guide to positioning, for why the ICP comes after differentiated value rather than before it.

• Full Funnel’s ICP for ABM, which publishes an actual Google Sheet.

• Allston Labs on closed-won deconstruction, the most rigorous small-sample method I found.

• Clozd on building a win-loss program, for the interview thresholds and why CRM loss reasons mislead.

• Clay’s account of what narrowing costs, which is the least comfortable line in the whole set: after they narrowed, “nearly all of our original customers churned.”

FAQ

What is an ideal customer profile? An account-level definition of which companies you are best at winning, serving, retaining and expanding, expressed as scored criteria and explicit disqualifiers rather than prose. It answers four questions: who will buy, who will buy smoothly, who can be implemented and serviced, and who will renew and expand.

How is an ICP different from a buyer persona? The ICP is the company, the persona is the person inside it. Qualify the account first, then map the buying committee. Treating them as synonyms produces a profile that can’t be used for routing or list building.

How is an ICP different from TAM? TAM is a dollar-value estimate of the whole opportunity. An ICP is a qualitative definition of which account types fit best. Every ICP account sits inside your TAM; most TAM accounts are not in your ICP.

How many customers do you need before you can define one? 5-7 closed-won deals support a provisional hypothesis, 30 is a stronger first pass, and 50 combined won and lost supports general pattern analysis. Below that, use the top 3 by revenue and top 3 by retention as a quarterly-revised hypothesis, with closed-lost as the control group.

How many attributes should an ICP have? Four to seven, and at least three. Past 8-10 dimensions you are over-specified and fitting noise, and the profile stops being usable by an SDR in real time.

How often should you update an ICP? Review monthly or quarterly against live closed-won data. Revise the operating definition only on a trigger: a new capability that changes who you can serve, retention or unit economics splitting by segment, a move upmarket, or behavioral data diverging from the original conception. Rewriting it every quarter creates whiplash in sales and marketing.

Who owns the ICP? One named owner with a change log, usually RevOps or whoever manages the CRM and the go-to-market cadence, with input from product, sales, marketing and customer success. At the earliest stage it belongs to the founders, because it is still being discovered in customer conversations.

Do you need a negative ICP? Yes, as its own section rather than the inverse of the positive criteria. Split hard exclusions you never target from soft ones you serve on inbound and referral only, and conditional ones a trigger can override. No-decision deals can reach 40% of a pipeline, and disqualifiers are what protect it.

Where should the ICP document live? Two places. A shared doc for the narrative version, and the CRM for the enforced version, as a fit score and a tier on the company object. If it only exists in the doc, it isn’t operating.

What is the difference between fit and intent? Fit is whether a company should buy from you, from stable attributes. Intent is whether it is looking right now, from recent signals. Score them separately and combine at routing: fit sets eligibility, intent and timing set the tier.

How do you know the ICP is working? Track ICP share of pipeline by count and dollars, win rate by tier, deal velocity by tier, and renewal rate by tier. If Tier 1 doesn’t convert materially better than Tier 3, the tiers are decorative.

When we incorporated GrowthX in 2025, we deliberately ran the work by hand first — a content marketing agency for clients like Ramp, Lovable, Vercel, Brex and Reddit — before building the product. That got us to 8 figure revenue and nearly 100 people.

In June 2026 we launched the first user-facing version of our product. Launching a new product inside a company that size is getting back to seed stage, but with a big team, customers, revenue and everything that comes with them.

A big engineering team lets us take bold steps. It also means I have to keep reminding myself that the team building this first version should behave like it’s seed stage.

I’ve kept rereading and referencing Paul Graham’s essays. And I wanted an audio version for my commute.

I pointed the same pipeline I built for a Computer History Museum oral history at Paul Graham’s blog to get the 230 essays I want to read again. Here is the first batch, grouped by theme; I will add all of the others over the next few days as they convert — sharing here in case others find it useful.

The audio is machine-narrated, and the text is adapted for listening rather than read verbatim. Every episode links to the original, which is the real thing.

Startups & Founders

Whole theme: 13 hours.

How to Get Startup Ideas

Startup ideas come from problems you already have, not brainstorming sessions. The test: would anyone use a crappy v1 from an unknown two-person team, the way a few thousand Altair owners needed Basic?
Duration: 40min, Publish Date: Nov 2012, Post: paulgraham.com/startupideas.html

Do Things that Don’t Scale

Startups take off because founders force them to, not because a better product attracts users on its own. Stripe’s founders set up beta users on the spot, and 10% weekly growth turns 100 users into 14,000 in a year.
Duration: 25min, Publish Date: Jul 2013, Post: paulgraham.com/ds.html

Startup = Growth

Growth rate is the one number that defines a startup, and it separates real ones from disguised small businesses. Track it weekly: 5-7% is good, 10% is exceptional, 1% means you haven’t found it yet.
Duration: 30min, Publish Date: Sep 2012, Post: paulgraham.com/growth.html

Schlep Blindness

Unconscious avoidance hides the best startup ideas, since founders flinch from tedious problems like payments and build recipe sites instead. Stripe fixed payments after a decade of hackers ignoring it; find your own version by asking what problem you wish someone else would solve.
Duration: 5min, Publish Date: Jan 2012, Post: paulgraham.com/schlep.html

Alive or Default Dead?

Most startups die from hiring too fast on mediocre growth, not from running out of ideas. Check whether current revenue growth gets you to profitability before the money runs out; Airbnb waited 4 months after Y Combinator to hire its first employee.
Duration: 8min, Publish Date: Oct 2015, Post: paulgraham.com/aord.html

What Happened to Yahoo

Yahoo had the money and the traffic to beat Google and blew it by calling itself a media company instead of hiring hackers. Programmers took orders from product managers instead of asking “what should we do,” and that killed the talent that would have caught search’s value before Google did.
Duration: 12min, Publish Date: Aug 2010, Post: paulgraham.com/yahoo.html

Ramen Profitable

Ramen profitable means a startup covers only the founders’ living costs, not a real business, but it ends dependence on investors. Google licensed search to Yahoo to get there; the danger is sliding into consulting and calling it a startup.
Duration: 11min, Publish Date: Jul 2009, Post: paulgraham.com/ramenprofitable.html

How to Start Google

Startups grow out of projects you build because you’re interested, not assignments you complete because you’re told. The test for a real idea: something you and your friends would be sad to lose, built by someone who’s spent years practicing a craft, before ever aiming at a company.
Duration: 15min, Publish Date: Mar 2024, Post: paulgraham.com/google.html

What Microsoft Is this the Altair Basic of?

Lame-sounding startups turn into giants; the trick is asking what company this small idea could become, not dismissing it on sight. One founder’s idea mapped to three separate possible Microsofts, varying by orders of magnitude.
Duration: 2min, Publish Date: Feb 2015, Post: paulgraham.com/altair.html

The 18 Mistakes That Kill Startups

Eighteen distinct mistakes kill startups, but they all funnel into one: not building something users want. Skip the sixth largest startup hub, skip Windows in the 90s, and take VC money only when you can spend it in weeks, not months.
Duration: 29min, Publish Date: Oct 2006, Post: paulgraham.com/startupmistakes.html

Change Your Name

Not owning the .com of your name signals weakness, even if mobile made domains matter less for actual traffic. Test: 100% of YC’s top 20 companies by valuation have it, only 66% of the current batch do.
Duration: 4min, Publish Date: Aug 2015, Post: paulgraham.com/name.html

Ideas for Startups

Startup ideas are worthless as standalone assertions; they only become real once you treat them as questions to test, not blueprints to follow. Y Combinator won’t fund single founders because good ideas emerge from friends arguing, not solo brainstorming.
Duration: 21min, Publish Date: Oct 2005, Post: paulgraham.com/ideas.html

Don’t Talk to Corp Dev

Corp dev only wants to buy you, and taking the meeting before you’re ready to sell wrecks morale and distracts you into a bad deal. Ask first: do you want to sell right now, not eventually?
Duration: 7min, Publish Date: Jan 2015, Post: paulgraham.com/corpdev.html

Mean People Fail

Meanness makes you stupid and costs you the best people, so the most successful founders win by building rather than fighting. Startups that get bought for money were never in it to change anything; the ones that keep going are.
Duration: 5min, Publish Date: Nov 2014, Post: paulgraham.com/mean.html

Relentlessly Resourceful

Getting from stuck to unstuck takes resourcefulness, not just persistence, since novel obstacles can’t be muscled through blindly. The test: ask if you’re relentlessly resourceful before starting a startup, or recruiting a cofounder.
Duration: 6min, Publish Date: Mar 2009, Post: paulgraham.com/relres.html

The Hardest Lessons for Startups to Learn

Release something minimal fast, then improve it constantly based on user reactions, because you never know your users well enough to guess right. Wufoo shipped its form builder before the database and got 83,000 users driving a car with no engine.
Duration: 26min, Publish Date: Apr 2006, Post: paulgraham.com/startuplessons.html

The Ronco Principle

Nice is not luck; it is the strategy that wins referrals in a transparent, unpredictable market. Ron Conway got Google, Facebook, and Twitter by being the one investor who never had to fake it.
Duration: 4min, Publish Date: Jan 2015, Post: paulgraham.com/ronco.html

What We Look for in Founders

Determination and flexibility matter more than raw brainpower; imagination matters more than speed at solving known problems. Airbnb’s founders paid rent selling Obama and McCain cereal before the “too crazy” idea worked.
Duration: 4min, Publish Date: Oct 2010, Post: paulgraham.com/founders.html

A Student’s Guide to Startups

Startups need less to succeed young than you’d think: what matters is stamina, cheap living, and co-founders you already know, not experience. Y Combinator’s official advice is to wait until after graduation anyway, because losing “student” as an excuse pushes you to try harder.
Duration: 35min, Publish Date: Oct 2006, Post: paulgraham.com/mit.html

Be Good

Making money is easier than making something great, so build the great thing first and profit follows. Craigslist and early Google prove it: Google ran ad-free for over a year and still won.
Duration: 15min, Publish Date: Apr 2008, Post: paulgraham.com/good.html

Before the Startup

Startups punish faking: you can talk your way past investors, but users only care if the product works. Don’t start one at 20 — build genuine expertise first, then let the idea find you, the way a Harvard student built VoIP just to call his girlfriend.
Duration: 25min, Publish Date: Oct 2014, Post: paulgraham.com/before.html

Can You Buy a Silicon Valley? Maybe.

Cities can buy a startup hub instead of growing one organically, if they pay startups enough to stay put. A million dollars each to 30 relocated companies is the cheapest real test.
Duration: 10min, Publish Date: Feb 2009, Post: paulgraham.com/maybe.html

Female Founders

Y Combinator funds more female founders than VCs do and helps them beat investor bias, the same tactic that worked for young founders a decade earlier. Current batch: 24% of companies have female founders, up from 9% two years ago.
Duration: 12min, Publish Date: Jan 2014, Post: paulgraham.com/ff.html

Five Founders

Steve Jobs, TJ Rodgers, Larry and Sergey, Paul Buchheit, and Sam Altman shaped how founders think about design, candor, hiring, and ambition. One line sums it up: better to make a small number of users love you than a large number kind of like you.
Duration: 4min, Publish Date: Apr 2009, Post: paulgraham.com/5founders.html

Founder Control

Founders keep majority board seats after a series A in a dozen YC-funded companies, following Zuckerberg and Pincus. Board control is not veto power over big decisions, but it means your view of shareholders’ interest wins when opinion splits.
Duration: 4min, Publish Date: Dec 2010, Post: paulgraham.com/control.html

Founder Mode

Founder mode says the standard advice to “hire good people and give them room” wrecks companies once you started them yourself. Steve Jobs ran skip-level retreats for Apple’s 100 most important people, regardless of rank on the org chart.
Duration: 7min, Publish Date: Sep 2024, Post: paulgraham.com/foundermode.html

Founders’ Accents

Accents are not the problem; being unintelligible is. Y Combinator ranked every company by valuation and found the highest-rated CEO with a strong foreign accent came in around 100th place.
Duration: 4min, Publish Date: Aug 2013, Post: paulgraham.com/accents.html

Frighteningly Ambitious Startup Ideas

Big startup ideas repel people before they attract them, which is why the best ones stay hidden. A new search engine only needs 10,000 hackers to start; email should become a todo-list protocol worth $1000 a month.
Duration: 21min, Publish Date: Mar 2012, Post: paulgraham.com/ambitious.html

Hiring is Obsolete

Undergrads are undervalued because investors can’t tell which 22-year-old is Bill Gates, and a failed startup at 22 costs nothing. Nine in ten startups fail, but Yahoo’s Zod Nazem still says he’d hire the guy whose startup tanked over the guy who took a safe job.
Duration: 26min, Publish Date: May 2005, Post: paulgraham.com/hiring.html

How Not to Die

Half of funded startups get rich just by not dying; the rest quit from discouragement, distraction, or grad school. Stay in contact, keep shipping, and never finish a sentence with “but we’re going to keep working on the startup.”
Duration: 10min, Publish Date: Aug 2007, Post: paulgraham.com/die.html

How to Start a Startup

Startups need three things: good people, something customers want, and low spending. Get all three right and you probably succeed; the “animal” test spots the first, and $10,000 seed money got Viaweb the third.
Duration: 53min, Publish Date: Mar 2005, Post: paulgraham.com/start.html

Learning from Founders

Startups do their best work looking unprofessional: no suits, no clean desks, no polished decks, just people building at 2am. The test for real productivity: could you cut off everything added just to look impressive, the way a car magazine stripped a “sports” car down to win the quarter mile.
Duration: 5min, Publish Date: Jan 2007, Post: paulgraham.com/foundersatwork.html

Microsoft is Dead

Microsoft stopped being scary around 2005, killed by Google, the web replacing the desktop, broadband, and Apple’s comeback. They still make money, like IBM, but no startup bothers inviting them to demo day anymore.
Duration: 7min, Publish Date: Apr 2007, Post: paulgraham.com/microsoft.html

Organic Startup Ideas

Startup ideas work best when they fix something broken in your own life, not something you guess strangers need. Test it against the Facebook-in-2004 case: nobody called a paper Harvard directory a startup idea, but it was one.
Duration: 6min, Publish Date: Apr 2010, Post: paulgraham.com/organic.html

Post-Medium Publishing

Publishers never sold content, only the format it came in: books, discs, paper. Better journalism didn’t cost more; The Economist ran 8.1 cents a page against Time’s 8.6, cheaper for the better read.
Duration: 10min, Publish Date: Sep 2009, Post: paulgraham.com/publishing.html

Startups in 13 Sentences

Make a few users love you, not many merely satisfied. The test: if you think you’re 85% there, ask how you’d know it isn’t 10%.
Duration: 8min, Publish Date: Feb 2009, Post: paulgraham.com/13sentences.html

The Airbnbs

Airbnb almost died before Y Combinator: a year of no growth, credit cards maxed out, one investor walking out mid-meeting. Three weeks into the batch, weekly fees went $460, $897, $1428 — ramen profitable at $4000/month, taped to their bathroom mirror.
Duration: 6min, Publish Date: Dec 2020, Post: paulgraham.com/airbnbs.html

The Cliffs Notes - Microsoft is Dead

Relevance leads revenue by five or ten years, so a company can be dead long before its balance sheet shows it. Technology companies almost never make comebacks; SAP still makes money, but nobody building new technology worries about it.
Duration: 2min, Publish Date: Apr 2007, Post: paulgraham.com/cliffsnotes.html

The Fatal Pinch

Losing money with runway shrinking and no revenue growth, betting on a raise that won’t come, is how startups die. Fire people, sell instead of code, or ask customers “what would you pay a lot for” instead of “will you buy this.”
Duration: 9min, Publish Date: Dec 2014, Post: paulgraham.com/pinch.html

The Other Road Ahead

Web-based software beats desktop software because it ships fixes and features instantly instead of once a year. Viaweb ran three to five releases a day, held under ten known bugs at any time, and cost about $5 in server hardware per user.
Duration: 68min, Publish Date: Sep 2001, Post: paulgraham.com/road.html

The Pooled-Risk Company Management Company

Startups sold for a lump sum and startups run for revenue are the same thing at different points on one curve; being profitable just guarantees you the average acquisition-market outcome instead of nothing. The proof: an acquirer is a “pooled-risk company management company,” and a public company will pay for yours if you can survive the wait, five years say.
Duration: 7min, Publish Date: Jul 2008, Post: paulgraham.com/prcmc.html

The Reddits

Steve and Alexis Reddit’s real idea was a cellphone fast-food app so bad Y Combinator rejected them, funding what became Reddit instead. It beat Slashdot by dropping human editors, launched in three weeks with a handful of users, and kept growing even through years of bad management.
Duration: 6min, Publish Date: Mar 2024, Post: paulgraham.com/reddits.html

The Trouble with the Segway

Riding a Segway makes you look smug, not efficient, because you stand still while it does the work. The fix: build one you ride foot-in-front like a skateboard, not standing like a sedan chair.
Duration: 2min, Publish Date: Jul 2009, Post: paulgraham.com/segway.html

Web 2.0

VC money for internet startups now goes toward getting bought, not IPOs, which is why this boom won’t repeat 1999. Rupert Murdoch’s $580 million Myspace purchase is only off from bubble math by a factor of 10.
Duration: 19min, Publish Date: Nov 2005, Post: paulgraham.com/web20.html

What I’ve Learned from Users

Startup problems repeat across companies, but founders rarely see their own clearly enough to rank them. YC’s summer 2012 batch broke at 80 startups on an O(n2) advising model, forcing a shift to sharded partner groups.
Duration: 12min, Publish Date: Sep 2022, Post: paulgraham.com/users.html

What Startups Are Really Like

Cofounders need character and commitment tested under stress, not raw ability, and everything takes 2-3x longer than expected. Persistence beats intelligence; launch the smallest possible version and let users tell you what’s wrong.
Duration: 28min, Publish Date: Oct 2009, Post: paulgraham.com/really.html

What the Bubble Got Right

Yahoo’s stock hit $200 on earnings partly generated by startups spending their own funding on Yahoo ads, an unintentional Ponzi scheme that still left a real $8 billion company after a 95% crash. Nerds win going forward: Google gets 82 million monthly users and $3 billion in revenue without ever running an ad.
Duration: 22min, Publish Date: Sep 2004, Post: paulgraham.com/bubble.html

Why It’s Safe for Founders to Be Nice

Nice founders don’t lose to rapacious ones because growth compounds and extraction rate is just a constant multiplier on the same curve. Extract half as much and you’re only 15 weeks behind after two years—optimize growth rate instead, and you pull ahead.
Duration: 5min, Publish Date: Aug 2015, Post: paulgraham.com/safe.html

Why Startup Hubs Work

Startup hubs work because everyone assumes founding a company is normal and useful strangers keep crossing your path. Sean Parker walking up to the Facebook founders in Palo Alto in 2004 is the proof.
Duration: 10min, Publish Date: Oct 2011, Post: paulgraham.com/hubs.html

Why There Aren’t More Googles

Startups that turn down acquisition offers usually do better, because the willingness to say no is itself the best predictor of success. Google and Facebook stayed independent only because Yahoo and Microsoft lowballed them.
Duration: 8min, Publish Date: Apr 2008, Post: paulgraham.com/googles.html

Why to Move to a Startup Hub

Startups do better the more of a startup hub they’re in, so all else equal, move to the biggest one. Facebook’s Boston VC passed and admits it “may turn out to have been a mistake.”
Duration: 8min, Publish Date: Oct 2007, Post: paulgraham.com/startuphubs.html

Why to Not Not Start a Startup

Half of the first Y Combinator batch got rich within two years, and zero percent regretted trying. The real filter isn’t age or experience: Sam Altman was funded at 19, and the fastest way to gain experience is to just start.
Duration: 33min, Publish Date: Mar 2007, Post: paulgraham.com/notnot.html

Why to Start a Startup in a Bad Economy

Recessions don’t sink startups; founders do, since technology adoption ignores the stock market. Microsoft’s Basic interpreter shipped into the 1975 recession, and cheap-to-run “cockroach” startups survive even when customers cut budgets one at a time.
Duration: 6min, Publish Date: Oct 2008, Post: paulgraham.com/badeconomy.html

Why TV Lost

Facebook killed TV: people wanted computers to connect with friends, not to watch shows, and piracy trained them to watch there anyway. Synchronicity and locality are dead, so local TV goes with them.
Duration: 9min, Publish Date: Mar 2009, Post: paulgraham.com/convergence.html

Why Twitter is a Big Deal

Twitter counts as a new messaging protocol, and those almost never appear or catch on. The last ones that stuck were TCP/IP, SMTP, and HTTP.
Duration: 1min, Publish Date: Apr 2009, Post: paulgraham.com/twitter.html

You Weren’t Meant to Have a Boss

Big companies force freedom down to zero as headcount grows; tree structures give each person freedom in inverse proportion to the whole org’s size. Groups of 8 work, 50 doesn’t, and Y Combinator founders visibly “grow inches taller” within three months of leaving one.
Duration: 14min, Publish Date: Mar 2008, Post: paulgraham.com/boss.html

Work, Ambition & Craft

Whole theme: 5 hours.

How to Work Hard

Talent and hard work aren’t substitutes; outliers need both, since you can’t change your talent, so doing great work reduces to working very hard. Bill Gates: “I never took a day off in my twenties. Not one.”
Duration: 17min, Publish Date: Jun 2021, Post: paulgraham.com/hwh.html

How to Do Great Work

Ability plus deep interest in something is the whole recipe; everything else is technique for sustaining it. Work in small, evolving versions rather than plans, and track a single test: are you working on what you most want to work on?
Duration: 63min, Publish Date: Jul 2023, Post: paulgraham.com/greatwork.html

Maker’s Schedule, Manager’s Schedule

Meetings cost programmers more than managers, because building anything takes half a day of unbroken time, not an hour. One meeting can break an afternoon in two pieces too small for real work, and even the prospect of it can flatten the morning before it happens.
Duration: 6min, Publish Date: Jul 2009, Post: paulgraham.com/makersschedule.html

The Other Half of “Artists Ship”

Every check an organization adds to prevent mistakes has a cost nobody weighs against the benefit, and it falls hardest on the best people, who simply stop showing up. Acquired startup engineers said they’d trade half the sale price just to release code same-day again.
Duration: 8min, Publish Date: Nov 2008, Post: paulgraham.com/artistsship.html

What to Do

Take care of people and the world, then make good new things. Newness is the test: a copy, however skilled, isn’t impressive, which is why Archimedes’s proof about the sphere and cylinder mattered more than repeating it would have.
Duration: 9min, Publish Date: Mar 2025, Post: paulgraham.com/do.html

The Anatomy of Determination

Determination beats intelligence at building a startup, and it splits into willfulness balanced by discipline, aimed by ambition. Push either force past the other and, like a squeezed melon seed, you spin off sideways instead of flying straight.
Duration: 9min, Publish Date: Sep 2009, Post: paulgraham.com/determination.html

Richard Hamming: You and Your Research

Great work takes years of compound effort, not luck; keep your office door open and spend Friday afternoons only on important problems. Ten to twenty problems you watch for, courage to attack them, and total commitment separate first-class scientists from the merely talented.
Duration: 67min, Publish Date: Mar 1986, Post: paulgraham.com/hamming.html

What Doesn’t Seem Like Work?

If a task other people treat as drudgery feels effortless to you, that’s the work you’re suited for. His father solved every textbook problem the day the book arrived, and Graham himself likes debugging.
Duration: 3min, Publish Date: Jan 2015, Post: paulgraham.com/work.html

A Project of One’s Own

Working on a project of your own beats working to order, even when it means longer hours and no clear payoff yet. Y Combinator judged applicants by their side projects, not their grades.
Duration: 14min, Publish Date: Jun 2021, Post: paulgraham.com/own.html

Cities and Ambition

Cities send one message that shapes ambition, and you absorb it just by walking around and overhearing conversations. Cambridge pushes intelligence, New York money, Silicon Valley power; picking the right one matters more than willpower, since even a Milanese-born Leonardo never became one.
Duration: 20min, Publish Date: May 2008, Post: paulgraham.com/cities.html

How to Do What You Love

Work isn’t supposed to hurt; the goal is finding what you’d do even without pay. Test it: would you still do the job for free while waiting tables to survive?
Duration: 24min, Publish Date: Jan 2006, Post: paulgraham.com/love.html

Disconnecting Distraction

Distractions evolve faster than your defenses; the internet replaced TV as the addiction because it still looks like work. Solution: a separate computer across the room for browsing, so the work machine can only be used to work.
Duration: 6min, Publish Date: May 2008, Post: paulgraham.com/distraction.html

Early Work

Fear of looking lame stops ambitious work before it starts, since early versions always look bad. Fix: treat new ideas as a challenge to your imagination, not a target for criticism, and overestimate their importance on purpose, the way Hardy said good professors do.
Duration: 14min, Publish Date: Oct 2020, Post: paulgraham.com/early.html

Earnestness

Interest in the problem itself, not money or fame, drives founders further than any other motive. Nerds get mocked for caring too much in high school; later, that same earnestness makes them the boss.
Duration: 10min, Publish Date: Dec 2020, Post: paulgraham.com/earnest.html

Good and Bad Procrastination

The most impressive people procrastinate on small stuff to work on big stuff, avoiding errands like laundry and email to get real work done. Ask what’s the best thing you could be working on, and why aren’t you.
Duration: 9min, Publish Date: Dec 2005, Post: paulgraham.com/procrastination.html

How to Lose Time and Money

Fortunes rarely get lost through spending, but through investments that don’t trip the “self-indulgent” alarm. Time follows the same pattern: fake work like email dodges that alarm while producing nothing, worse than a wasted day on the sofa.
Duration: 4min, Publish Date: Jul 2010, Post: paulgraham.com/selfindulgence.html

Six Principles for Making New Things

Simple solutions to overlooked problems win because they look unimpressive enough that nobody else bothers to compete for them. Viaweb shipped under 10,000 lines of code with no credit card processing, and Reddit launched with no design at all.
Duration: 7min, Publish Date: Feb 2008, Post: paulgraham.com/newthings.html

Superlinear Returns

Returns for performance are superlinear, not linear: half as good means zero customers, not half the customers. Growth compounds through exponential growth and thresholds, so track growth rate, not absolute numbers, and follow curiosity over careerism.
Duration: 25min, Publish Date: Oct 2023, Post: paulgraham.com/superlinear.html

The Right Kind of Stubborn

Persistence and obstinacy look alike from outside but run on different engines: one is attached to the goal, the other to its first idea. The tell is response to pushback: the Collison brothers listen “with predatory intensity,” the obstinate just glaze over.
Duration: 12min, Publish Date: Jul 2024, Post: paulgraham.com/persistence.html

The Top Idea in Your Mind

Whatever you’re not actively controlling becomes what you think about in the shower, and money and disputes hijack that slot faster than anything else. Newton tried to quit publishing entirely after wasting years defending his color theory against critics like Linus.
Duration: 6min, Publish Date: Jul 2010, Post: paulgraham.com/top.html

When To Do What You Love

Following your interests only pays off at the extremes: barely at all, or wanting to get rich. Bill Gates loved writing software for customers, not just programming; Apple, Google, and Facebook all started as side projects.
Duration: 8min, Publish Date: Sep 2024, Post: paulgraham.com/when.html

Writing & Language

Whole theme: 2 hours.

Writes and Write-Nots

Soon almost no one will be able to write, because AI removes the pressure that once forced people to learn. That matters because writing is thinking: “if you’re thinking without writing, you only think you’re thinking.”
Duration: 3min, Publish Date: Oct 2024, Post: paulgraham.com/writes.html

The Best Essay

The best essay tells you something surprising about the most important topic possible, but real timelessness requires the opposite: writing about things that never get culturally absorbed, like kids overdoing things or adults lying to kids.
Duration: 23min, Publish Date: Mar 2024, Post: paulgraham.com/best.html

Good Writing

Sentences that sound bad usually carry ideas that are wrong, because fixing the awkward rhythm forces you to fix the thinking underneath it. The test: rewrite a paragraph to be one line shorter, and it always gets better, never worse.
Duration: 9min, Publish Date: May 2025, Post: paulgraham.com/goodwriting.html

Writing, Briefly

Writing well matters because it generates ideas, not just conveys them. Draft fast, rewrite until 80% of the ideas emerge after you start, and cut without mercy.
Duration: 3min, Publish Date: Mar 2005, Post: paulgraham.com/writing44.html

The Need to Read

Writing teaches you how to think, not just how to record thoughts you already had. Complicated, ill-defined problems get solved by writing about them; easy ones don’t need it.
Duration: 2min, Publish Date: Nov 2022, Post: paulgraham.com/read.html

Writing and Speaking

Good ideas carry writing but barely matter in speaking, where delivery and audience reactions dominate. One test: transcribe a great talk and you’ll often find, as Graham did with a famous speaker, that after ten sentences there’s nothing there.
Duration: 6min, Publish Date: Mar 2012, Post: paulgraham.com/speak.html

Write Like You Talk

Written language is more work to read and gives you the false feeling you’re saying more than you are. Test: would you say this sentence to a friend, or does it clank?
Duration: 4min, Publish Date: Oct 2015, Post: paulgraham.com/talk.html

How to Write Usefully

Useful writing states claims as strongly as possible without going false, then cuts anything not sure enough to survive: no exceptions, no hedged filler. One test: don’t publish a sentence unless you’d bet on it, the way Robert Morris won’t say anything unless it’s worth hearing.
Duration: 16min, Publish Date: Feb 2020, Post: paulgraham.com/useful.html

Putting Ideas into Words

Writing exposes gaps in thinking that feel complete until you try to put them into words. Read your own draft as a stranger would, and fix whatever it flags as missing.
Duration: 4min, Publish Date: Feb 2022, Post: paulgraham.com/words.html

The Age of the Essay

Real essays start with a question, not a thesis, and let the answer surprise you along the way. Track what surprises you, since the surprising 1% is where the truth usually hides.
Duration: 26min, Publish Date: Sep 2004, Post: paulgraham.com/essay.html

The List of N Things

Lists of n things read easier because their structure is explicit and their points are independent, so a bad one costs you nothing. That ease is also the trap: you can’t have a new idea bigger than the slot you pre-built for it, like a 5-paragraph essay capping thought at three points.
Duration: 7min, Publish Date: Sep 2009, Post: paulgraham.com/nthings.html

The Shape of the Essay Field

Writing for smart people about important things means writing for the young, since they have the most room left to change their minds. The Selfish Gene rewired how Graham saw evolution; that’s the kind of surprise to aim for.
Duration: 4min, Publish Date: Jun 2025, Post: paulgraham.com/field.html

Write Simply

Simple sentences and ordinary words let ideas reach readers with less friction, so more people read to the end. The test: if a sentence feels clumsy rather than fancy, cut it.
Duration: 3min, Publish Date: Mar 2021, Post: paulgraham.com/simply.html

Ideas, Thinking & Learning

Whole theme: 3 hours.

How to Disagree

Disagreement has a hierarchy, from name-calling up to refuting someone’s central point. The test: can you quote the exact passage you think is wrong and say why?
Duration: 9min, Publish Date: Mar 2008, Post: paulgraham.com/disagree.html

How to Think for Yourself

Correct isn’t enough; original ideas only pay off when most people think you’re wrong, whether you’re a scientist, an investor, or a startup founder renting airbeds on strangers’ floors.
Duration: 21min, Publish Date: Nov 2020, Post: paulgraham.com/think.html

Crazy New Ideas

Never dismiss an implausible idea proposed by a reasonable domain expert; ask why instead of arguing, since Copernicus published heliocentrism in 1532 and it took until the mid-1600s to win over astronomers.
Duration: 8min, Publish Date: May 2021, Post: paulgraham.com/newideas.html

Keep Your Identity Small

Political and religious arguments go nowhere because both let anyone opine without expertise, and both get absorbed into identity. Test it on programming languages: X-vs-Y debates turn religious the moment people call themselves “X programmers.”
Duration: 5min, Publish Date: Feb 2009, Post: paulgraham.com/identity.html

The Four Quadrants of Conformism

People sort into four types by conformism and aggression, and the aggressively conventional ones—tattletales as kids, mobs as adults—cause most of the trouble. Test: only the independent-minded can tell how much of the idea-space has actually been banned.
Duration: 12min, Publish Date: Jul 2020, Post: paulgraham.com/conformism.html

What You (Want to)* Want

Wanting isn’t fully free: you control what you do, but not what you want, and barely what you want to want. Three or four “want to”s in, change becomes nearly unimaginable, like a string of 9s approaching 1.
Duration: 3min, Publish Date: Nov 2022, Post: paulgraham.com/want.html

Being a Noob

The more of a noob you feel locally, the less ignorant you actually are globally — moving to a new country or field makes you feel dumber while making you know more. The discomfort is a hunter-gatherer instinct built for stable problems, not for cryptocurrency.
Duration: 2min, Publish Date: Jan 2020, Post: paulgraham.com/noob.html

Two Kinds of Judgement

Selecting a national team of 20 doesn’t require judging the 20th player accurately, only that the roster ends up good. Most rejections work the same way, so stop treating admissions or hiring decisions like a court case demanding fairness.
Duration: 4min, Publish Date: Apr 2007, Post: paulgraham.com/judgement.html

A Way to Detect Bias

You can spot bias without seeing the applicant pool: if a selection process discriminates against a group, that group’s survivors will outperform everyone else. First Round Capital’s own portfolio showed female-founded startups beating the rest by 63%.
Duration: 3min, Publish Date: Oct 2015, Post: paulgraham.com/bias.html

Alien Truth

Physics and math would hold for aliens too, but so would less obvious things like Occam’s razor and the value of controlled experiments. The test for a real philosophical truth: would an intelligent alien accept it too?
Duration: 4min, Publish Date: Oct 2022, Post: paulgraham.com/alien.html

Beyond Smart

Being smart and having new ideas are different things, and most people conflate them. Test: an obsessive interest in one topic and independent-mindedness matter more than raw intelligence for finding something new.
Duration: 8min, Publish Date: Oct 2021, Post: paulgraham.com/smart.html

Fashionable Problems

Working problems everyone agrees are played out hides the most territory left to explore, because talented people avoid looking unfashionable. Try essays, Lisp, or venture funding: find a new angle into a “done” field and the payoff scales with its size.
Duration: 1min, Publish Date: Dec 2019, Post: paulgraham.com/fp.html

How to Be an Expert in a Changing World

Beliefs about a changing world go stale, and experts are usually just experts on an earlier version of it. Bet on people, not ideas: earnest, energetic, independent-minded founders like Airbnb’s got funded despite the idea looking bad.
Duration: 6min, Publish Date: Dec 2014, Post: paulgraham.com/ecw.html

How to Do Philosophy

Words break down when pushed too far, which is why philosophy chases unanswerable questions instead of useful ones. The fix: judge an idea by whether it makes anyone do something differently, the way “controlled experiment” or “evolution” already do.
Duration: 27min, Publish Date: Sep 2007, Post: paulgraham.com/philosophy.html

How to Get New Ideas

New ideas come from noticing anomalies at the edge of what you know, not from brainstorming. Get close enough to a field’s frontier and the gaps become obvious; chase one and it can grow into a whole new area.
Duration: 1min, Publish Date: Jan 2023, Post: paulgraham.com/getideas.html

How You Know

Forgetting most of what you read doesn’t mean it was wasted: it compiles into your mental models even after the source vanishes. Reread important books anyway, since the same text compiles differently at different points in your life.
Duration: 4min, Publish Date: Dec 2014, Post: paulgraham.com/know.html

Is It Worth Being Wise?

Wisdom is a high average across situations; intelligence is high peaks in a few. Confucius’s sage stayed calm because his problems had a correct answer, but a mathematician who invents things goes to bed discontented four nights out of five.
Duration: 20min, Publish Date: Feb 2007, Post: paulgraham.com/wisdom.html

Mark Twain: Corn-pone Opinions

A man’s opinions come from where his bread and butter is, not from independent reasoning. Test it against hoopskirts and historical novels: styles vanish and spread by imitation, never by argument.
Duration: 11min, Post: paulgraham.com/cornpone.html

The Bus Ticket Theory of Genius

Great work needs a pointless-seeming obsession, not just talent and grit. Darwin couldn’t stop cataloging species, Ramanujan filled slates with series for no reason, and Tolkien built an elvish language before writing a word of fiction.
Duration: 15min, Publish Date: Nov 2019, Post: paulgraham.com/genius.html

The Risk of Discovery

Famous scientists took real risks that biographies erase by only remembering the bets that paid off. Newton split his time between physics, alchemy, and theology, all equally promising at the time; only one worked.
Duration: 1min, Publish Date: Jan 2017, Post: paulgraham.com/disc.html

What Charisma Is

Charisma comes from liking people, not performing for them: uncharismatic politicians smile so voters will like them, charismatic ones smile because they already like the crowd. Clinton reaching over his own Secret Service agents to shake hands is the tell; wanting contact makes people want you back.
Duration: 3min, Publish Date: Nov 2004, Post: paulgraham.com/recharisma.html

Why Smart People Have Bad Ideas

Founders waste months on their first startup idea because it felt cool, not because customers would pay for it. Ask early: of everything we could build, is this the one people will pay most for?
Duration: 18min, Publish Date: Apr 2005, Post: paulgraham.com/bronze.html

Since starting GrowthX, I haven’t had time to read much other than Slack and PRs. The only thing I can consume these days are audiobooks. Everything else piles into a queue of articles and PDFs I keep telling myself I’ll get to.

A lot of that queue is Computer History Museum oral histories. I’ve watched plenty of the video ones, but some of the best interviews are transcript only.

So I finally coded the workflow I’d been meaning to build forever. It cleans up the transcript, splits the speakers, gives each of them their own voice, and narrates it with Deepgram Aura-2, with chapter marks so you can skip around.

The first one I wanted to listen to was Hartmut Esslinger, the founder of frog design and a design legend. After earning their into becoming Apple’s design partner by winning Apple’s Snow White competition and defined the look of Apple hardware until Jony Ive’s translucent iMacs. The Computer History Museum has a 28-page PDF, which came out as 85 minutes of audio across 20 chapters.

Audio Version of Esslinger’s Oral History 👇

Some highlights from my notes

Winning Apple’s Snow White prize by refusing the brief

Apple ran Snow White in 1982 to unify a product line that had grown by accident. Esslinger was running a small studio out of a village in the Black Forest, and got in the door through a party at a former intern’s house.

The assumption inside Apple was that the answer looked like Sony, a company he had spent a decade consulting for. He said no. “That’s Japanese. This is different. America must be different.”

What he pitched instead was “simple, white, innocent, sexy, and also a bit more radical, a new global brand.” His argument for why Apple could get away with it: “There was no real market for the product, none at all, so Apple can do what is right.”

Where he thought Apple’s own designers had gone wrong: “Olivetti never had a design language. Olivetti had a design philosophy.”

He won the competition, then told the in-house team they couldn’t implement it, because “the way you do it, you will destroy it in the first second.” Most of them left.

The two million dollar design fee was a manufacturing argument

Two million dollars a year from Apple, starting in 1982, for a studio of eight people. The highest fee ever paid to a designer at the time, and he defends it with unit economics rather than craft.

“At a million computers you save 100 bucks per computer, that’s 100 million dollars. So what is a design fee compared to that?”

The savings were engineering. Apple had been painting its plastic enclosures. “You have a piece of plastic, five dollars. You paint it, it’s 15.” So frog moved them to unpainted Lexan, coloured all the way through.

Then zero-draft moulding. Moulded parts normally need slightly angled walls so they can be pulled out of the mould, and that taper is what makes plastic look cheap. He had solved it at Sony with a collapsible core and 16 or 32 mould pieces instead of two or four, which also put more toolmakers in parallel and shipped the part sooner. Cheaper, faster, and it looked like machined steel.

The Apple account still wasn’t profitable for frog at first: “you have to prove what you can do.”

Every argument he makes for design in three hours is a manufacturing or P&L argument.

A one percent royalty, not fees, is what built frog design

In 1974, before Apple, KaVo hired frog to redesign a dental chair. “It’s all hostile; it looks like an electric chair. Why not make it friendly?”

The CEO loved the work and had no money. They were building about 20 units a year. So Esslinger asked for a royalty instead of a fee. KaVo offered one percent, he took it, and the contract ran to less than a page.

A year later, at a regional trade show, they sold 200-plus by 11am on the opening day and stopped taking orders because the factory couldn’t make them. He redesigned it for volume. It sold for more than twenty years at roughly two million a year in license fees.

That royalty, not the Apple fee, is what paid for the model shop, an early CAD system, and salaries high enough that industrial designers were paid like engineers.

A hundred ideas to Steve Jobs, five shipped

Richard Sapper designed for IBM for decades, and later did the ThinkPad. Esslinger quotes him: “I bring them a thousand ideas and they take one. And I try a hundred times and maybe they do it once. What a waste. What a waste.”

His own version at Apple: “we took a hundred ideas to Steve and we made five good ones and co-developed them.” Fifty times the throughput, same calibre of designer.

Why in-house teams go nowhere, using Motorola as the example: good designers, but one reported into engineering and another into marketing, with no connection between them, so “nobody is really held accountable for corporate vision.”

And on the companies who ask to be the next Apple: “everybody wants to be like Apple, but when you tell them what it takes — budget, perfect models, realistic simulations, absolutely no shortcuts and a huge love for people — they balk.”

How frog design doubled its team without losing itself

Winning Snow White came with a condition: an office in Silicon Valley, near Apple. frog was eight people in a converted garage in the Black Forest, baking models in the kitchen oven.

So they hired eight more and had each existing person mentor their exact counterpart for three months, role for role. Then they split the doubled team and moved half of it to Campbell, California. After that, people rotated between Germany and California every month, administration included. “I was really careful not to lose the culture.”

He treated a forced doubling as a chance to get better rather than something to survive: “the challenge is great, but we have to get better.”

Nobody was hired into a new office. The office was cloned, then moved. I’ve done org-doubling worse than this.

Thinking big and acting small

“People have to think big, but they have to act small.”

The last thing he says, after three hours.

To me, a pattern through the interview is the idea of never settling on one altitude. Pitching Apple on a new global brand, then arguing about the taper on a moulded wall, all in the same conversation.

Core

Traditional software has bugs. LLM software has behaviors.

You don't test behaviors - you observe them, measure them, and guide them. This requires a fundamentally different approach.

Three Modes of Observation

Mode 1: Introspection The model evaluates itself during generation. Like a writer reviewing their draft before hitting send. Built-in confidence scoring, explanation generation, self-critique loops. This happens at inference time, costs tokens, but provides immediate guardrails.

Mode 2: Instrumentation Automated measurement of what actually happened. Response times, token usage, embedding similarities, retrieval success rates. Like application performance monitoring but for AI quality. Runs continuously, at scale.

Mode 3: Interpretation Humans make sense of what matters. Not everything can be measured - sometimes you need human judgment on tone, helpfulness, or edge cases. Expensive but irreplaceable for subjective quality and ground truth.

What This Looks Like in Practice

During Development:

• Start with exploration, not test cases - explore what your model can do

• Capture interesting behaviors as you discover them (the good, bad, and weird)

• Write expectations, not assertions: "should be empathetic" not "must contain 'sorry'"

• Build your eval suite backwards - get it working first, then define what "good" means based on real outputs

During Deployment:

• Shadow scoring on real traffic before switching

• Gradual rollout based on confidence thresholds

• Automatic fallback to previous version if scores drop

• No binary deploy/rollback - it's a confidence dial

During Operation:

• Stream of observations, not error logs

• Distribution of scores, not pass/fail counts

• Anomaly detection, not threshold alerts

• Behavioral drift tracking, not uptime monitoring

During Improvement:

• Human labels on strategic samples, not random QA

• Disagreement cases between evaluators get priority

• Corrections become test cases automatically

• Fine-tuning happens on production-validated examples

Embracing LLM Constraints

Embrace Uncertainty: Stop pretending LLMs are deterministic. Design for confidence intervals, not binary outcomes.

Compose, Don't Compile: Evaluators should be Lego blocks - mix and match for your use case. Need safety + factuality + tone? Stack them.

Optimize for Learning: Every evaluation should make the system smarter. If it doesn't feed back into improvement, why measure it?

Human Time is Sacred: Only escalate to humans when machines disagree or confidence is low. Make their input count by turning it into reusable signals.

That survey was the most complete map of the field, and it was drawn before reasoning models and native tool calling existed. Those two changes, plus a run of 2025-2026 benchmarks, answered the question a taxonomy can’t: which of the 58 still earn their tokens.

The bigger change isn’t in the taxonomy at all. In 2024 prompting was a specialist skill. You wrote prompts if you were building an AI product. Everyone else shipped software without touching one. In 2026 every programmer writes prompts all day.

So this is a pass back through the research: what the newer studies measured, and where those numbers contradict advice that still gets repeated. Almost every claim is paper-backed or straight from provider documentation, linked inline.

What changed

• Example count. Then: more exemplars improve performance, with diminishing returns past 20. Now: it depends on the task. Classification climbs past 250, reasoning saturates at 4-125 then degrades, and one example often scores below zero.

• Zero-shot. Then: the fallback when you have no examples. Now: the default. On GSM8K, 8 exemplars bought nothing over zero-shot.

• Example labels. Then: whether demonstrations must be strictly valid is unclear. Now: settled. Random labels cost 0-5%; format and label space do the work.

• Example order. Then: order can swing accuracy from below 50% to over 90%. Now: unchanged and still unfixed. Demonstrations at the start are worth up to 6 points.

• Chain-of-thought. Then: a core reasoning technique. Now: math and symbolic only. 95% of the MMLU gain came from questions containing “=”.

• “Let’s think step by step”. Then: task-agnostic, works anywhere. Now: deprecated on reasoning models by all three vendors, at 20-80% more response time.

• Role prompting. Then: can sometimes improve accuracy on benchmarks. Now: no reliable accuracy gain on current models, and irrelevant persona detail can cost 30 points.

• Self-consistency. Then: improves arithmetic, commonsense and symbolic reasoning. Now: collapsed on frontier models, at +1.6 points for 15x the tokens.

• Tree of Thoughts. Then: a decomposition technique. Now: still works, at 5-100x the tokens. Batch jobs only.

• Temperature and top-p. Then: yours to tune. Now: reasoning models reject them outright.

• Agents and tool use. Then: not covered. Now: the loop is an API contract, not a prompt pattern.

• Writing the prompt. Then: by hand. Now: optimizers beat reinforcement learning on most tasks, if you have evals.

What prompt engineering is

The working definition is OpenAI’s: “the process of writing effective instructions for a model, such that it consistently generates content that meets your requirements.” The word doing the work is consistently. Getting one good answer out of a model is easy. Getting the same quality on the next thousand inputs is the job.

The smallest version of the whole discipline, from the Prompt Engineering Guide:

Classify the text into neutral, negative or positive.
Text: I think the vacation is okay.
Sentiment: neutral
Text: I think the food was okay.
Sentiment:

Output: neutral. One line of instruction, one worked example, one input. The model already knows what sentiment is. The example is there to stop it answering in a paragraph.

Context engineering is the wider frame. Anthropic’s line is that prompt engineering covers “methods for writing and organizing LLM instructions for optimal outcomes,” while context engineering covers curating everything else that lands in the window at inference time: retrieved documents, tool definitions, memory, history. Karpathy’s version is “filling the context window with the right information for the next step.” For a single-turn feature the prompt is most of the game. Inside an agent it’s a subset.

Prompting is part of the development cycle now

A SKILL.md for Claude is a prompt. So is the CLAUDE.md at the root of a project, the description on every tool you hand an agent, every subagent definition in a Claude Agent SDK pipeline, the goal you hand a loop, and the .prompt files in Output. None of those are chat. They ship, they get reviewed, and they break on upgrade.

Which is the part people still underrate. A 2025 migration case study found prompts stabilized for GPT-4-32k passing at only 98% on GPT-4.1 and 97.3% on GPT-4.5-preview. Prompts rot like dependencies, and nothing in CI notices unless you wrote the eval.

Anatomy, and the one part that’s a trust boundary

Instruction with a definition of done, context, delimited input, output schema. That decomposition is old news, and OpenAI’s reasoning guidance still puts it well: “avoid vague instructions by defining what counts as done.” Here it is with all four parts labelled:

# Identity                          <- role, one sentence
You are a support triage assistant for a B2B billing product.

# Instructions                      <- instruction, with a definition of done
Assign exactly one category from the list. Output the category name only,
lowercase, no punctuation. If the ticket matches none, output "other".

# Categories                        <- context the model can't infer
billing_dispute | refund_request | plan_change | access_issue | other

# Ticket                            <- the input, delimited
<ticket>
Charged twice for the October invoice, need one reversed.
</ticket>

Output: billing_dispute. Those delimiters around the ticket are doing real work. They tell the model where your instructions stop and a stranger’s text begins.

The part worth arguing about is where each piece goes. System and developer messages carry higher privilege in the instruction hierarchy OpenAI trains models to respect. Role, rules and output format go there. Untrusted content goes in user messages or tool results, never in a developer message: OpenAI’s agent safety guidance is explicit about not putting untrusted variables there. The message-role split is an access-control decision, not formatting.

Ordering is the part that stopped being portable. Anthropic says to put long documents at the top, above your query, and claims it “improves performance across all models.” Google says the same: all context first, question last. OpenAI’s general guide puts context near the end, but the GPT-4.1 guide recommends instructions both before and after a long document. Three vendors, three answers. Pick per model and measure.

Role prompting got quietly demoted

“You are an expert X” was sold as an accuracy technique. It isn’t one any more, and the vendors have moved without announcing it. OpenAI’s o3 guidance scopes role prompting to “setting the base behavior, tone and outlining the set of actions that are possible.” Anthropic’s Claude Opus 4.6 tutorial goes further: “Skip the role setting... It infers the appropriate level of expertise from the task and context you provide.”

The research is why. A Wharton study found no expert or low-knowledge persona that reliably improved GPQA Diamond across six models, and no significant gain on MMLU-Pro for five of six. Peer-reviewed work found expert personas usually neutral, and models “highly sensitive to irrelevant persona details, with performance drops of almost 30 percentage points.” A 2026 preprint measured a librarian persona dropping mean correctness on Claude Opus from 0.92 to 0.67.

Keep the role sentence for tone and scope. Don’t expect it to make the answers more correct.

The prompt-writing checklist

When a prompt underperforms, the cause is usually somewhere on this list and not in the technique.

1. State the task and what “done” looks like. Not “summarize this” but “three bullets, under 20 words each, no preamble.”

2. Say what to do, not only what to avoid. Negative instructions aren’t testable.

3. Delimit every piece of input with XML tags, Markdown headings, or """. Pick one convention and hold it.

4. Put the output format in the system or developer message, then enforce it at the API with structured outputs where the schema matters.

5. One role sentence, task-specific. No stacked adjectives.

6. Static content first, variable content last so prompt caching can hit the prefix.

7. Show it to a colleague with no context. Anthropic’s golden rule: if they’d be confused, so is the model.

The foundational techniques, re-scored

Zero-shot is the default, not the baseline

A peer-reviewed 2025 paper put Qwen2.5-72B-Instruct on GSM8K at 95.83 zero-shot with corrected answer extraction, against 95.75 with 8-shot exemplars. The examples bought nothing.

One example is the count most likely to make things worse. GPT-4o-mini on AG News scored 0.8446 F1 at zero-shot, dropped to 0.8248 at one shot, and only passed its own baseline at two (arXiv:2607.22969). GPT-5 on biomedical QA does the same thing: 0.762 zero-shot, 0.737 at one shot, 0.768 at five (arXiv:2509.04462). If you shipped a one-shot prompt without testing zero-shot, you may have paid tokens to lose accuracy.

Start zero-shot with an explicit output format. Add machinery when your evals show a gap, not in advance.

Few-shot examples are versioned config

Two to five labeled pairs, for patterns that are easier to show than to describe: a house style for summaries, an odd label taxonomy. Anthropic is the only vendor that publishes a number, and it’s “3-5 examples for best results.” OpenAI says “a handful.” Google says experiment, and to “always include few-shot examples.”

What examples teach is narrower than most people assume. Replace the gold labels in your demonstrations with random ones and accuracy falls 0-5% absolute. You can watch it happen:

This is awesome! // Negative
This is bad! // Positive
Wow that movie was rad! // Positive
What a horrible show! //

Output: Negative. Correct, despite every label above it being wrong. The examples transmitted the label space and the // format, not the answers. Strip the format and you do worse than no examples at all.

The fragility is the story. The accuracy gap between demo-selection algorithms reached 45% on MRPC. Moving demonstrations from the front of a prompt to the back flipped over 30% of QA predictions in a 2025 study, without improving correctness. Google warns that too many examples cause overfitting, and examples buried mid-prompt hit the “lost in the middle” positional problem.

Three inputs that all look like cosmetics (which examples, how many, where) move accuracy more than most model swaps do. Formatting alone is in the same range: researchers who changed nothing else measured GPT-3.5-turbo differences of up to 40% on code translation. Version the examples, diff them, cover them with evals.

Two rules with numbers behind them. Put demonstrations at the start: best position across ten model families, worth up to 6 points, and it puts the stable text where the cache can reach it. And never sort examples by label: at 1,169 shots on Clinic-150 that cost 25.7 points.

Chain-of-thought is mostly a math-formatting effect

The technique arrived twice. The first version needed eight hand-written reasoning chains per task, which is real authoring work for one prompt. Four months later, most of that gain turned out to be free: five words appended to the question, Let’s think step by step. On text-davinci-002 that took MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%, with no examples at all.

Their Figure 1 is still the clearest before-and-after:

A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?

Without the trigger the model answers 8. It halves once and stops. With it: “There are 16 balls in total. Half of the balls are golf balls. That means that there are 8 golf balls. Half of the golf balls are blue. That means that there are 4 blue golf balls.”

CoT pays when the answer needs two or more steps that depend on each other, and answering directly collapses them into one.

A meta-analysis of more than 100 papers put it at +14.2 pp on symbolic reasoning, +12.3 pp on math, and no meaningful gain on commonsense tasks. 95% of CoT’s total MMLU gain came from questions containing “=” in the question or output.

Where it works, it’s decisive. +59.9 pp on GSM8K for Llama-3.1-8B-Instruct, and Qwen-2.5-7B-Instruct went from 23.1% to 91.1% on GSM8K in the same 2026 study.

Where it doesn’t, it costs you. The same Qwen model dropped 28.7 points on HumanEval code generation. Forcing explicit steps cut o1-preview accuracy by 36.3% absolute on implicit-learning tasks. On reasoning models, prompted CoT bought small GPQA gains for 20-80% more response time, and 35-600% longer responses on non-reasoning models.

Use it for math and symbolic work on non-reasoning models. On reasoning models, use the native control instead: Gemini 3 guidance says to replace CoT prompting with thinking_level: 'high' and simplified prompts.

One thing to know before you show a reasoning trace to anyone. Causal mediation across twelve models found GPT-4 changed its answer only 30% of the time when its reasoning chain was swapped for a perturbed one. CoT changes accuracy; it does not give you an audit trail.

The advanced techniques, and what they cost

Tree of Thoughts belongs in batch jobs

Tree of Thoughts branches several candidate thoughts per step, scores them, and searches with BFS or DFS. Game of 24 with GPT-4: 74% success against 4.0% for CoT.

The appendix prices it at 0.47 for best-of-100 CoT, and the authors say it “could require 5-100 times more generated tokens than CoT” depending on the search setup. My judgment, not a paper’s: batch jobs on search-like problems, planning and constrained generation. I’ve never found a user-facing path where the latency was acceptable, and I don’t have a controlled benchmark behind that.

ReAct survived as a loop, not as a parser

ReAct interleaves Thought → Action → Observation: 71% task success on ALFWorld against 45% for action-only prompting.

In 2026 the loop is an API contract. The assistant turn carries the thought as text and the action as a structured block:

{
  "stop_reason": "tool_use",
  "role": "assistant",
  "content": [
    { "type": "text",
      "text": "I'll check the current weather in San Francisco for you." },
    { "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": { "location": "San Francisco, CA", "unit": "celsius" } }
  ]
}

You run the tool and send the observation back as a user message, keyed to the call:

{ "role": "user",
  "content": [{ "type": "tool_result",
    "tool_use_id": "toolu_01A09q90qw90lq917835lq9",
    "content": "15 degrees" }] }

Failures go back the same way with "is_error": true rather than being swallowed; the model recovers better from a visible error than from a silent one. Loop while stop_reason is tool_use, exit on end_turn. Anthropic’s docs never say “ReAct”; the mapping is structural. If you are still regexing Action: out of free text, that’s a 2023 artifact.

The Thought half is the contested part. A peer-reviewed 2025 study found performance “is minimally influenced by the interleaved reasoning trace.” StateAct scored 27.80% on WebShop with the thought step removed against ReAct’s 17.80%. When2Tool is the alarming one: adding Reason-then-Act dropped Llama-3.1-8B tool accuracy from 79.5% to 31.2%.

It cuts the other way too. Anthropic’s think tool with tuned prompting hit 0.584 on τ-bench Airline against a 0.332 baseline, beating native extended thinking at 0.412. ToolGT found structured templates beat free-form CoT which beat no thought at all.

Both results can be true. Models trained for native tool calling already know what to call, so an extra verbal layer is at best neutral. Text-formatted agents still benefit. Test with the trace and without it rather than assuming.

Self-consistency has collapsed on frontier models

Self-consistency samples several CoT paths at nonzero temperature and takes the majority answer. On PaLM-540B and GSM8K that went from 56.5 to 74.4 at 40 paths, with 5-10 paths capturing most of it. Early-stopping variants matched 40-sample accuracy on GSM8K at an average of 14.65 samples.

Now run it on a frontier model. Gemini-2.5-Pro on MATH-500 goes from 98% at one path to 99.6% at 15. A 1.6-point gain for roughly 15× the tokens.

RAG is the cheap option, not just the grounded one

The grounding case is well known: in the XRAG benchmark, GPT-4o went from 6.30% accuracy without retrieval to 75.40% with oracle retrieval.

The cost case gets skipped. The Token Tax study priced long-context prompting at ~4.50 for semantic RAG on the same manufacturing benchmark. Roughly 26× for stuffing the window instead of retrieving into it.

One failure mode to measure for: a 2026 Springer benchmark found hybrid retrieval sometimes increased hallucination in smaller models even as retrieval recall improved. Score faithfulness, not just recall.

Let the optimizer write it

The newest option here is to stop writing the prompt yourself.

GEPA runs a candidate prompt on a minibatch, captures the traces and whatever the environment said back (compiler errors, judge notes), and shows a reflection model the whole tuple so it can propose a revision. It keeps a Pareto frontier of candidates that win on at least one example rather than one global best. On GPT-4.1 Mini across six benchmarks it beat MIPROv2 by more than double: +12.19 aggregate against +5.64. Against GRPO reinforcement learning it won five of six tasks, by up to 19 points, using up to 35× fewer rollouts.

Prompt evolution now beats reinforcement learning on most tasks at a fraction of the sample cost, which was not true in 2024.

The prerequisites are the catch, and they’re the same three things evals need: a metric that scores an example, a representative dataset, and a test set you haven’t touched. Without them it optimizes noise. Across 72 optimization runs on Claude Haiku 4.5, 49% scored below zero-shot: a coin flip, driven by eval sets too small to measure with.

Three more things worth knowing before you reach for it. More data is not better: Decagon found peak performance at 20 samples, with 500 costing 2% accuracy and 75% prompt bloat. The reflection model has to be strong, since a weak one returns your seed prompt nearly unchanged. And it optimizes around bugs rather than fixing them. GEPA drove a defective GSM8K seed from 23.81% down to 13.50% while the root cause sat there untouched.

We haven’t put this in production at GrowthX. Our prompts change faster than a compile cycle would pay for, which is a statement about our release cadence rather than about GEPA.

Sampling settings: mostly stop tuning them

OpenAI. Temperature 0-2, no stated default, no top_k support. Reasoning control is reasoning_effort.

Anthropic. Temperature 0.0-1.0, default 1.0, top_k documented for advanced use only. Reasoning control is effort, the adaptive-thinking parameter.

Google Gemini. Temperature 0.0-2.0, default 1.0, top_k supported and fixed at 64 on Gemini 2.5 Pro. Reasoning control is thinkingLevel or thinkingBudget.

On classic models the advice is unchanged: 0-0.3 for extraction, classification and function calling, 0.7-1.0 for creative generation, and alter temperature or top_p “but not both.” OpenAI’s cookbook uses temperature=0 for function calling and 0.2 for structured summarization.

On reasoning models, the knobs are being taken away. Anthropic’s Claude Opus 5 / Sonnet 5 generation returns a 400 error on non-default temperature, top_p, or top_k; you steer with the adaptive-thinking effort parameter. Google’s Gemini 3 guidance recommends keeping temperature “at its default value of 1.0” and warns that lowering it can cause looping and degraded reasoning. On reasoning models there is close to nothing left to tune.

The caveat that bites people writing tests: Anthropic documents that “even with temperature of 0.0, the results will not be fully deterministic,” and Google says the same. Exact-match assertions on model output will flake.

Prompting inside an agent loop

Single-turn prompting is one shot at getting the wording right. Agent prompting has to survive dozens of iterations, each one putting tool output back into the window.

• Tool descriptions are prompt engineering. Anthropic calls them “by far the most important factor in tool performance” and asks for 3-4 sentences minimum. The evidence backs the emphasis: renaming functions to func_N significantly drops performance, and tools with edited descriptions get over ten times more usage. OpenAI’s version is the intern test: could a human use this function given only what you gave the model? Say when to use a tool and when not to; hedged instructions like "If in doubt, use [tool]" cause overtriggering.

• Make invalid states unrepresentable. Enums for finite sets, user_id not user, documented ranges. Turn on strict mode, and note it disables parallel tool calls.

• Narrow the mandate per step. OpenAI’s Atlas hardening guidance says to “avoid overly broad prompts like ‘review my emails and take whatever action is needed.’” Broad mandates compound errors across steps and widen the injection surface.

• Curate state, don’t accumulate it. Decide at each step what history and retrieval results enter the window, and filter tool output the same way.

• Tool output is untrusted input. Anything a tool fetched from a third party is attacker-controlled text arriving inside your loop.

Frameworks are checklists, not engineering

Three have published provenance. CLEAR (Leo S. Lo, Journal of Academic Librarianship, 2023) is Concise, Logical, Explicit, Adaptive, Reflective, and works as a review rubric. PROMPT Design (Sarah Hartman-Caverly, Penn State, 2024) is a fill-in-the-blanks scaffold for non-engineers; the academic AI PROMPT framework covers similar ground with more emphasis on delimiting elements and token length. PASTAS (Lo again, portal: Libraries and the Academy, 2026) was written for the agentic era, and its Tools and Safeguards items map onto the two sections above.

They help a team remember requirements. They don’t replace measuring: a CLEAR-compliant prompt that fails your evals is still a failed prompt.

Fine-tuning is still the last resort

Every provider now documents the same sequence: evals, prompting, RAG for external knowledge, fine-tuning when prompting hits a measured ceiling. OpenAI’s model-optimization guide says “the prompt engineering process may be all you need,” and its fine-tuning platform is winding down, with existing users able to create training jobs only “for the coming months.”

The thresholds worth holding to:

• Volatility. Microsoft’s split is dynamic content → RAG, stable content → fine-tuning. Hamel Husain’s version: “fine-tuning works best to learn syntax, style and rules whereas RAG works best to supply the model with context or up-to-date facts.”

• Example count. OpenAI recommends 50 well-crafted demonstrations for SFT; a peer-reviewed 2025 study puts the classification break-even near 100 labeled examples. At 20-40, Jason Liu argues you are better off with prompt caching.

• Good enough. The applied-llms.org rule: “if prompting gets you 90% of the way there, then finetuning may not be worth the investment.”

• The bill nobody quotes. Husain estimates “99% of the labor involved with fine-tuning is assembling high-quality data.” The LIMA result cuts both ways: 1,000 curated examples fine-tuned a 65B model to outputs preferred over GPT-4’s in 43% of comparisons. Quality beats volume, and curating 1,000 excellent examples is still real work.

Prompt caching is the middle path in production. Anthropic prices cache reads at 0.1× base input and AWS says caching can cut costs up to 90%. At GrowthX our source material changes weekly, so we’ve stayed on prompting plus RAG. I haven’t hit a case where a fine-tune was worth the data-assembly cost, which is a statement about our volatility rather than about fine-tuning.

Security: assume the prompt leaks and the tools lie

The documented record includes 15+ named injection incidents between 2022 and 2026, and in 2026 alone five CVE identifiers across four agent-framework incident groups. Google measured a 32% relative increase in malicious prompt-injection categories on the web between November 2025 and February 2026.

Injection. Riley Goodside demonstrated it publicly against GPT-3 in September 2022. By 2025 it was zero-click: EchoLeak (CVE-2025-32711) exfiltrated data from Microsoft 365 Copilot with no user interaction. Researchers reported successful injections in 56% of 144 tests across 36 models. Wording will not save you; the mitigations are architectural. Anthropic’s guardrail guidance says to deliver third-party content inside tool_result blocks, keep it out of system prompts, and JSON-encode it so delimiters are unambiguous. OpenAI’s agent guidance adds that defense “cannot rely only on filtering inputs. It also requires designing the system so that the impact of manipulation is constrained.” Least privilege per tool, confirmation before irreversible actions.

Leaking. Kevin Liu pulled Bing Chat’s “Sydney” instructions in February 2023. OWASP’s guidance is that “the system prompt should not be considered a secret, nor should it be used as a security control.” Keep credentials and keys out of the prompt; assume it leaks.

Jailbreaking. From DAN to Best-of-N attacks at 89% success on GPT-4o with 10,000 augmented prompts. Layered classifiers are the answer that measures: Anthropic’s Constitutional Classifiers cut jailbreak success from 86% to 4.4%. Google Cloud’s Model Armor screens prompts and responses, OpenAI’s free Moderation API covers the basics.

OpenAI calls adversarial robustness “a hard, open problem,” and Anthropic notes that even a 1% attack success rate “still represents meaningful risk.” Budget for defense in depth from day one if your agent reads email or the open web, both channels the Gemini promptware research weaponized.

Where prompts usually break

• Negative framing. “Be concise” is not testable. Anthropic recommends the positive form: instead of "Do not use markdown", write "Your response should be composed of smoothly flowing prose paragraphs."

• No output schema. It breaks parsers, and per OpenAI’s safety docs structured outputs also “eliminate freeform channels that attackers can exploit.” One constraint, two problems.

• Trusting the documented context window. Performance degrades 13.9%-85% within claimed windows, with relevant information missed in middle positions. We hit it ourselves: a vendor documented 128,000 tokens and our task started degrading well before that. Retrieve less, better.

• Technique for technique’s sake. CoT cost o1-preview 36.3 points on implicit-learning tasks, and in a 2026 medical study it cut accuracy 5.7% while few-shot examples cut it 11.9%.

• Not reading what your framework sends. Husain’s prompt inspection found LangChain’s SmartLLMChain shipping a typo ('Let'w') in its critique prompt and Guidance making 7 API calls where 2 sufficed. Log the raw request once per integration, minimum.

Evals are the job

Draft, run against a fixed test set, score, change one variable, version like code. The loop exists because output varies: even temperature 0 isn’t deterministic, so one passing run proves nothing.

This is the reasoning behind Evals Driven Development: repeated runs scored against a rubric instead of exact-match assertions. After we rewrote the rubric, the measured pass rate on our 60-query test set went from 74% to 86%. That number is ours; the method transfers. Shreya Shankar’s guidance to review at least 100 traces before designing any rubric matches our experience: rubrics written before error analysis test failures that never happen.

The checklist:

1. Collect 50-100 real inputs before writing the prompt. Include the ugly ones.

2. Simplest zero-shot prompt with an explicit output format, run 3-5 times per input.

3. Error analysis first, rubric second, written from observed failures.

4. One variable at a time: wording, examples, format, technique.

5. Version every prompt, log every run. PromptLayer’s registry documents version diffs and release labels; LangSmith and Langfuse do the same job. Ours lives in Output, where prompts and evals sit next to traces and cost tracking in the repo. Disclosure: Output is my open-source project.

6. Gate merges on eval regressions. DeepEval runs in pytest with an --official baseline; the pattern generalizes.

7. Track tokens and cost per prompt version next to quality. A quality win that doubles spend is a decision, not a detail.

Calibrate LLM judges against human labels before trusting them, and score asynchronously where you can. Synchronous judging adds seconds per request.

FAQ

How do I pick a technique? Zero-shot with an output format. Few-shot when the pattern resists description. CoT for math and symbolic work on non-reasoning models, native thinking controls on reasoning models. Self-consistency or ToT only for batch work where an eval shows a gap worth 5-100× the tokens. RAG when the knowledge is external, changes often, or is cheaper to retrieve than to stuff.

How many examples should I use? Anthropic says 3-5 and is the only vendor publishing a number. The rule that matters more: measure zero-shot and 2-5 shots before you commit, because one example often scores below none.

What temperature should I use? 0-0.3 for extraction and function calling, 0.7-1.0 for creative work, one of temperature or top_p but not both. On reasoning models, leave it alone and use reasoning_effort, effort, or thinkingLevel.

Should I tell the model it’s an expert? For tone and scope, yes, one sentence. For accuracy, no. No vendor has published a controlled ablation showing it works on a current reasoning model, and irrelevant persona detail has measured costs.

When does fine-tuning beat prompting? When prompting has a measured ceiling, the target is a stable style or format, and you have 50-100+ curated examples plus appetite for the data work. Below that, caching and RAG are cheaper.

Should I use an automatic prompt optimizer? Only if you already have an eval suite, a few hundred representative examples, and a held-out test set. With those, GEPA is the strongest thing available. Without them, half the runs come out worse than zero-shot.

Minimum security posture for a prompted feature? Assume the system prompt leaks, so keep secrets out of it. JSON-encode untrusted input and keep it out of developer messages. Least privilege per tool, a classifier on inputs, confirmation before irreversible actions.

Is prompt engineering a viable career in 2026? No, and it never was. It was a job title minted by a hype cycle. The University of Oulu study found 72 exact-title postings out of 20,662, under 0.5%.

The skill went the other way. Lightcast counted mentions rising from ~1,400 in 2023 to ~6,300 in 2024, and O’Reilly measured 456% year-over-year growth in prompt engineering content usage. It became part of the job, like knowing how to write a migration.

“Loop engineering” is the 2026 reprise of the same bs. Real skill, fake title. Learn it as part of the stack.

The 2024 version of this post aged in two years. This one will too, and the part that ages first is always the benchmark numbers rather than the method. If you run these against your own eval set and get different numbers, I want to hear about it. Mine are one team’s.

What we’ve achieved in just ~7 months since inception is remarkable. $3M+ in ARR. Profitable. Grew team to 15. Zero funding!

Here’s the backstory:

At the beginning of the year, I took a sabbatical to immerse myself in the world of large language models (LLMs). I dove deep into retrieval-augmented generation (RAG), prompting strategies, fine-tuning techniques, and more.

Fast forward to mid-September, I got introduced to Marcel Santilli. Within minutes, it was clear—he was onto something big and had the experience + ambition to deliver on it.

Marcel's pitch:

• Brands need to become publishers.

• Founders need to become influencers.

• You need strategy + experts to deliver it, but AI can make it scalable.

Having previously helped with marketing before, I immediately got it—and Marcel had already proven the model’s effectiveness. The technical and product needs for the business were exactly the kinds of things I had spent the last year+ studying.

In the past four months, we’ve spent long hours validating a roadmap, getting the right team structure, and defining the GTM. Now we have the ambitious goal of reaching $15 million in ARR by 2025. To achieve this, my top priorities are building the right infra and putting together a great technical team.

We just opened two roles for founding engineers: https://growthx.ai/careers. We are a remote-first company hiring from anywhere, as long as we can have a 4-hour timezone overlap with San Francisco. If you’re interested, don’t hesitate to reach out!

Summary

The main idea is that every single company that doesn't rely on cheating (corruption, government ties, etc.) to come to life and achieve long-term success (measured in profit/revenue) has to have at least one or more of these "powers":

1. Scale Economies

2. Network effect

3. Counter positioning

4. Branding

5. Switching costs

6. Cornered resources

7. Process

Power progression

The idea is that these powers aren't constant either, and especially in tech, things are much more dynamic than static. Just because you succeed with one of these powers doesn't mean it will sustain forever, and each power has different defensibility in the face of competition as well. He qualifies the impact and the moment you can rely on and try to chase each power based on the phases of the business:

1. Origination

2. Take-off phase

3. Stability phase

Each phase is self-explanatory, but his idea is that chasing specific barriers in the right phase of the business will build much higher competitive advantage and barriers for new players. Also, he is explicitly focused on business phases, not product life-cycle.

Powers applicable to Origination Phase

Phase where you are between the creation of the business, building everything and just launched and/or haven't achieved growth or market fit yet.

• Counter positioning

• Cornered resources

Applicable to Take-off Phase

Take-off phase is when the business starts to achieve fast growth.

• Scale economies

• Switching Costs

• Network effect

Applicable to Stability phase

The break between take-off and stability is when unit growth falls below about 30%-40% year.

• Branding

• Process power

Now to the description and examples of each power (in order of business phases):

Counter positioning (origination)

Can be identified by:

• An upstart who developed a superior business model.

• That business model's ability to successfully challenge well-entrenched incumbents.

• Steady accumulation of customers, all while incumbent remains paralyzed and unable to respond.

Ex: The new business model is superior to the incumbent's model due to lower costs and/or the ability to charge higher prices.

Barrier of entrance:

Incumbent(s) will eventually ask: "Am I better off staying the course, or adopting the new model?" If the answer is "stay off", then the barrier is high. A newcomer adopts a new, superior business model which the incumbent does not mimic due to anticipated damage to their existing business.

Counter-Positioning versus Disruptive Technologies (Clayton Christensen)

• Digital photography vs Kodak - this is a DT, but not CP.

• In-N-Out vs. McDonald's - this is CP, but not DT

• Netflix streaming vs. HBO via cable - this is both CP and DT

The case for playing humble: Cognitive Bias can play a role in deterring the incumbent. But the challenger, by its posture, may be able to influence such a move. How to attempt this? In its ascendancy, the challenger should avoid the temptation of trumpeting its superiority, instead suppressing that urge and adopting a tone of respect toward the incumbent.

Counter positioning is not exclusive. Different from other powers like Network Effect, strong counter positioning doesn't require (or create) a winner-takes-all situation.

Five stages of the incumbent when dealing with successful counter positioning from a new challenger:

1. Denial (aka, "we are not competitors")

2. Ridicule (aka, "we are better")

3. Fear

4. Anger

5. Capitulation (frequently too late)

Blockbuster execs showed all 5 stages in their public statements when facing Netflix. I personally dealt with founders on 1 and 2 many times (and always end up on 5 later).

Once market erosion becomes severe, a Counter-Positioned incumbent comes under tremendous pressure to do something; at the same time, they face great pressure to not upset the apple cart of the legacy business model. A frequent outcome of this duality? Let's call it dabbling: the incumbent puts a toe in the water, somehow, but refuses to commit in a way that meaningfully answers the challenge.

CP often underlies situations in which the following happens at the same time:

For the challenger:

• Rapid share gains

• Strong profitability (or promise of it)

For the incumbent:

• Share loss

• Inability to counter the entrant's moves

• Eventually management shake-up(s)

• Capitulation

Cornered resources (origination)

Benefit:

Cornered Resource can emerge in varied forms, offering uniquely different benefits. It might, for example, be preferential access to a valuable patent, such as that for a blockbuster drug; a required input, such as a cement producer's ownership of a nearby limestone source, or a cost-saving production manufacturing approach.

Ex: In the Pixar case, the Brain Trust produced an uncommonly appealing product—"superior deliverables"—driving demand with very attractive price/volume combinations in the form of huge box office returns.

Barrier:

Barrier in Cornered Resource is unlike anything we have encountered before. You might wonder: "Why does Pixar retain the Brain Trust?" Any one of this group would be highly sought after by other animated film companies, and yet over this period, and no doubt into the future, they have stayed with Pixar. In the case of spin casting technology, it is patent law, and in the case of cement inputs, it is property rights.

The Five Tests to validate Cornered Resource:

1. Individual/Exclusive: If a firm repeatedly acquires coveted assets at attractive terms, then the proper strategy question is, "Why are they able to do this?" Ex: if Exxon was able to persistently gain the rights to desirable properties, then understanding their path to that is key. Do relative scale allows them to develop better discovery processes? If so, their discovery processes are the Cornered Resource.

2. Non-arbitraged: cost to keep the resource should be low enough to afford consistent differential returns (ex: a movie hiring Brad Pitt might move the box office but won't necessarily 10x the cost).

3. Transferable: If a resource creates value at a single company but would fail to do so at other companies, then isolating that resource as the source of Power would entail overlooking some other essential complement beyond operational excellence.

4. Ongoing: In searching for Power, a strategist tries to isolate a causal factor that explains continued differential returns. Ex. Post-it notes patent lasted for 30 years

5. Sufficient: The final Cornered Resource test concerns completeness: for a resource to qualify as Power, it must be sufficient for continued differential returns, assuming operational excellence. Ex. George Fisher joining Kodak wasn't enough.

Switching costs (take-off phase)

Ex: SAP - 43% of users are unhappy but 85% say they will continue to pay for the product, and the sales validate their statements.

Benefit:

A company that has embedded Switching Costs for its current customers can charge higher prices than competitors for equivalent products or services. Benefit only possible when you have additional offerings to sell.

Barrier:

To offer an equivalent product, competitors must compensate customers for Switching Costs. The firm that has previously roped in the customer, then, can set or adjust prices in a way that puts their potential rival at a cost disadvantage, rendering such a challenge distinctly unattractive.

Types of switching costs:

• Financial: $ cost of investing and purchasing new solutions (ex: SAP)

• Procedural: loss of familiarity or risk of adoption of a new product (ex: "nobody was ever fired for choosing IBM")

• Relational: Affection with the provider (ex: sales team) or product/identity as a user.

Network Effect (take-off phase)

Benefit:

Network Economies occur when the value of a product to a customer is increased by the use of the product by others (Branch Out vs LinkedIn)

A company in a leadership position with Network Economies can charge higher prices than its competitors, because of the higher value as a result of more users.

Barrier:

For Network Economies, the barrier is the unattractive cost/benefit of gaining share, and this can be extremely high. In particular, the value deficit of a follower can be so large that the price discount needed to offset this is unthinkable.

Example: The value of LinkedIn's HR Solutions Suite comes from the numbers of LinkedIn users, so LinkedIn can charge more than a competing product with fewer participants.

Characteristics:

Industries exhibiting Network Economies often exhibit these attributes:

1. Winner take all: Businesses with strong Network Economies are frequently characterized by a tipping point: once a single firm achieves a certain degree of leadership, then the other firms just throw in the towel. Game over—the P&L of a challenge would just be too ugly. For example, even a company as competent and with as deep pockets as Google could not unseat Facebook with Google+.

2. Boundedness: As powerful as this Barrier is, it is bounded by the character of the network, something well-demonstrated by the continued success of both Facebook and LinkedIn. Facebook has powerful Network Economies itself but these have to do with personal not professional interactions. The boundaries of the network effects determine the boundaries of the business.

3. Decisive early product: Due to tipping point dynamics, early relative scaling is critical in developing Power. Who scales the fastest is often determined by who gets the product most right early on. Facebook's trumping of MySpace is a good example.

Scale Economies (take-off)

The quality of declining unit costs with increased business size is referred to as Scale Economies.

Benefit:

Some condition which yields material improvement in the cash flow of the Power wielder via reduced cost, enhanced pricing and/or decreased investment requirements.

Barrier:

Some obstacle which engenders in competitors an inability and/or unwillingness to engage in behaviors that might, over time, arbitrage out this benefit.

For Scale Economies, the Benefit is straightforward: lowered costs. In the case of Netflix, their lead in subscribers translated directly into lower content costs per subscriber for originals and exclusives.

Example:

Netflix paid $100M for House of Cards and their streaming business had 30M customers, then the cost per customer was three dollars and change. In this scenario, a competitor with only one million subscribers would have to ante up $100 per subscriber.

This situation creates a very difficult position for Netflix's smaller-scale streaming competitors. If they offer the same deliverable as Netflix, similar amounts of content for the same price, their P&L will suffer. If they try to remediate this by offering less content or raising prices, customers will abandon their service and they will lose market share. Such a competitive cul-de-sac is the hallmark of Power.

Branding (stability phase)

Branding as power to charge higher prices and sustain market share.

Ex: Purchased a Diamond ring at Tiffany for $16,600 and one of similar size and cut at Costco for $6,600, then asked a reputable gemologist and appraiser to assess the rings' values: Costco ring at $8,000 plus setting costs, more than $2,000 above the selling price. Assessed Tiffany ring at $10,500 plus setting costs at a non-brand-name retailer.

"You got exactly what they said you were getting. Anything that is brand-name and has developed a reputation that Tiffany has developed, they've earned it over the years for quality control. You can go there [and] you don't have to think twice about your purchase. And you pay for that."

Benefit:

A business with Branding power is able to charge a higher price for its offering due to one or both of these two reasons:

1. Affective valence - The built-up associations with the brand elicit good feelings about the offering, distinct from the objective value of the good. (Ex Tiffany's)

2. Uncertainty reduction - A customer attains "peace of mind" knowing that the branded product will be just as expected. Consider another example: Bayer aspirin. Search for aspirin on Amazon.com and you will see a 200 count of Bayer 325 mg. aspirin for $9.47 side-by-side with a 500 count of Kirkland 325 mg. aspirin for $10.93. So Bayer has a price per tablet premium of 117%. Some customers still would prefer the Bayer because of diminished uncertainty: Bayer's long history of consistency makes customers more confident that they are getting exactly what they want.

Note that the Benefit from Branding does not depend on prior ownership, as with Switching Costs.

Barrier:

A strong brand can only be created over a lengthy period of reinforcing actions (hysteresis), which itself serves as the key Barrier.

Process (stability phase)

Ex. Toyota Process (TPS)

Benefit:

A company with Process Power is able to improve product attributes and/or lower costs as a result of process improvements embedded within the organization. For example, Toyota has maintained the quality increases and cost reductions of the TPS over a span of decades; these assets do not disappear as new workers are brought in and older workers retire.

Barrier:

These process advances are difficult to replicate, and can only be achieved over a long time period of sustained evolutionary advance. This inherent speed limit in achieving the Benefit results from two factors:

1. Complexity - Returning to our example: automobile production, combined with all the logistic chains which support it, entails enormous complexity. If process improvements touch many parts of these chains, as they did with Toyota, then achieving them quickly will prove challenging, if not impossible.

2. Opacity - The development of TPS should tip us off to the long time constant inevitably faced by would-be imitators. The system was fashioned from the bottom up, over decades of trial and error. The fundamental tenets were never formally codified, and much of the organizational knowledge remained tacit, rather than explicit. It would not be an exaggeration to say that even Toyota did not have a full, top-down understanding of what they had created —it took fully fifteen years, for instance, before they were able to transfer TPS to their suppliers. GM's experience with NUMMI also implies the tacit character of this knowledge: even when Toyota wanted to illuminate their work processes, they could not entirely do so.

Innovation is the key to achieving any of the powers

Innovation is achieved by creating compelling value. There are three distinct paths to creating compelling value.

1. Capabilities-Led Compelling Value: Adobe Acrobat. Here the key capability brought to bear was Adobe's existing fluency at the intersection of software and graphics.

2. Customer-Led Compelling Value: Corning Fiber Optics. Interaction at a distance. The uncertainty in this case is technical: "Can we invent it?"

3. Competitor-Led Compelling Value: the Sony PlayStation. Sony perceived the value in immersion of 3D graphics and the gap in the competitors' offerings (Sega/Nintendo) to double down on 3D and launch PS1.

Now, I'm recharged and excited to be back!

Here are some learnings, a new project I shipped this week, and some plans for the next few months:

Aimap.today: An Experiment with Agents & Flows

I recently focused on learning about agent tooling to build an AI company researcher for personal use.

The plan: Create an agent to find funding news, identify AI-related companies, analyze their websites, build a knowledge base, write a long form article about it, and categorize them into three tiers (foundational AI tech, AI-enabled, and AI users).

I explored several agent frameworks: CrewAI, LangGraph (LangChain’s agent take), LlamaIndex's Llama-Agents (a microservice take on agents, where each agent is it’s own service), and AgentOps (very cool instrumentation tool). I also tried building flows with AirOps and n8n. In the process of fetching and enriching the data I also tried Jina Reader/SERP, You.com API, Apollo API, and Clay agent.

Ultimately, for now, I decided to take a stab at coding my own thing. Most existing frameworks focus on LLMs talking to each other, but all I needed was a workflow/pipeline to chain multiple processes together.

My current boilerplate is in Ruby, and I wasn't satisfied with the existing packages for API clients like Claude and OpenAI. So, I coded these from scratch as well. The whole project, including study time, took about two weeks.

👉 The result is: https://aimap.today . Please, check it out and let me know what you think.

I built this for myself and plan to add new companies weekly. There's more I want to do, including a weekly digest for newly researched companies. If you sign-up you’ll be notified when I finish the weekly digest part.

My next step is to explore more out-of-the-box platforms like AirOps and n8n, and try to covert my little workflow to these tools.

Code-wise the pipeline currently looks like this:

Joined Techstars for the First San Francisco class

I'm thrilled to share that I’ll be helping Techstars’ first San Francisco class as an EIR (Entrepreneur in Residence). I've been following my friend and former boss, Neal, as he ran Techstars Chicago and Oakland. When he mentioned the new San Francisco class, I jumped at the chance to participate.

The cohort is impressive, featuring many interesting AI-enabled businesses. I'm excited to work with all the founders. You can view the full cohort here: https://www.techstars.com/newsroom/techstars-announces-inaugural-class-of-san-francisco-accelerator

It's great to be back. Building aimap.today has been an a cool learning experience, and I'm eager to continue improving and expanding it.

Joining Techstars as an EIR is another exciting opportunity I'm looking forward to.

I also may be joining a team on a new project soon, but I'll save that for future updates.

Going forward, my goal is to resume weekly updates.

Three types of play

Started considering which spaces are interesting now. I think this take from Hamilton Helmer is a good one: Three types of play:

1. The tech play: example of semiconductors, you get Intel and etc.

2. The first-tier play: companies that wouldn't exist before semiconductors (Microsoft)

3. The usage play: car companies with tons of semiconductors. They already existed, just got better.

The whole video is worth watching, but heres’ the time-stamp about AI.

I'm personally more interested in the second play. And finding these companies early and identifying which ones are already in a good direction is so hard.

Brainstorm exercise

A good place to start could be looking at common roles and how they will be impacted and can be augmented or even replaced.

A programmer's job is already seriously impacted, customer support even more so. Same for SDRs. Some other roles, like localization, have already been pretty much replaced. These highly impacted roles are the things I'm most interested in.

So, just as a thought exercise toying with Python + Langchain, I wrote a 4-step chain with Claude 3.5 to do following:

1. Brainstorm departments and roles common across any vertical

2. List the most common tasks in those roles

3. Brainstorm ideas for how each could be augmented or automated

4. (Rank them by difficulty and confidence, but this was just for nudging the model)

I only spent a couple of hours and $3 in credits, but the results turned out better than I was expecting.

There's obviously some nonsense and some things that are regular ML models more than LLMs in the output (my fault, I should have optimized the last step of the chain better), but a lot of solid ones that I think haven't happened yet, and some others in the “augmentation” category fully in progress like Figma, Intercom, Cursor, GitHub Copilot, etc. If you squint and look at the roles with the best ideas, you can see Piper, Fin, Harvey, Wisq, Devin, 11x.ai, etc

You can see the whole list here:

https://docs.google.com/spreadsheets/d/1vpXzfUFNhIZJp9CEmiZqGKaVrP_Xx8McMZZlLCxiHNg/edit?gid=0#gid=0

Next step, I guess, is to start collecting companies and maybe have a little agent help me with the research.

—

I haven't had time to think through the whole list yet, but here...Some of the ones that stood out that the ideas are mostly very doable, or already happening:

Traditional machine learning

When you move beyond simply calling APIs, you'll encounter the traditional ML stack and terminology. For instance, if you need to adjust your embeddings and choose a different model, you'll need a solid grasp of metrics like BPB, BPC, Perplexity, and so on. The same goes for evaluating production data.

I also prefer not to have components in a stack that I don't understand at a high level, even if I don't have to work with them directly (for example, if I'm an EM on the product side or a product manager).

For ML theory I’m currently half way through this course. It may be a few years old, but I appreciate how practical and real-world focused it is. If anyone has other suggestions, I'm very open to them.

I can't recall the exact source of this slide (I think it was from Hamel Hussain’s talk on AI Engineer conf this month) , but I believe it provides a reasonable overview - and my goal is to have at least a basic understanding of everything under the horizontal line.

Evaluations

Last week, I began studying evaluations more seriously for two reasons. First, I'm working on understanding the various stages of the development cycle. Second, I need to establish a better workflow for iterating on our AI bot at Canopy.

For theory around evals I really liked Chip’s third chapter of her WIP book. This post about Evaluations from Hamel Husain is also so good, and this one as well.

This is would be the ideal cycle for me:

At Canopy, our tech stack is entirely Ruby, and our AI Assistant was built using direct API calls to OpenAI. While I did implement some observability, I still need a more robust way to monitor the quality of answers in production and a solid test suite that I can run when updating the model or modifying the prompts.

Ideally, I would have automated tests similar to regular test units that I can run whenever I'm iterating on prompts, models, or content format for the rag. Once things are deployed, I'd like to have comparable quality metrics for the production data, with alerts if quality drops below certain thresholds and proper visibility.

There are quite a few LLM-focused evaluation tools available, and many of them are really solid. I don't mean to dismiss their work, but for my specific context, here's my current assessment:

1. They do too much

   1. Most tools attempt to offer solutions for all four stages of my diagram, but these companies are often small, seed-level teams building in a rapidly evolving space. Many of these products feel either half-baked or are still trying to figure out their business model. I'd prefer a tool that focuses solely on the two evaluation parts (dev & production) and doesn't try to handle the prototyping and deployment aspects.

2. Their business models are still unclear

   1. Many tools are in the typical early-stage B2B mode, lacking a product-led path and aiming to lock users into annual contracts with case-by-case pricing. I understand this is important in a new market with early-stage products, but I'd love to see more product-led options (especially for Canopy's context, a small team where the bot isn't their core feature yet).

3. Vendor lock in / dependency risk

   1. To enable tracing, many tools heavily wrap around API clients for services like OpenAI, and some don't even offer public API versions. Tools that offer "solutions" for the deployment part often proxy calls under their own API, so instead of calling OpenAI directly, you call their API. This introduces a third-party proxy as a potential failure point, in addition to OpenAI/Anthropic/etc., and it's built by an understaffed seed company.

4. Pretty much no solutions for non-Python/JavaScript stacks

   1. Most niceties for automatic logging/tracing and basic metrics like latency come from their SDK, which are often limited to Python/JS. While understandable given the early stage, most tools also don't offer a public API. For more elaborate setups, a reasonable workflow could be to follow the traditional ML path of using Python for the stack and wrapping it as an API for other languages to call, instead of relying on a proxy maintained by a tiny company in a crowded space. It's a bit of a bummer because for AI products that only need LLMs, it's an overcomplication in the stack. You'll definitely need query augmentation, retries, rate-limiting controls, and other things a product-level software already handles, and now you'll need to duplicate that in the Python stack too. It's one thing when you're making an API around a model you control, but it's another when you're making an API around another API.

If you're already using a Python or JS stack, you have fewer things to worry about. However, for Canopy (Ruby stack), the situation is a bit different. I think I'll go with the following approach:

• First: Find a tool that has a public API support for sending run data for production, build a small Ruby client for that, and do production level monitoring only (similar to how’d use a service like Sentry). Ignore all the other 3 stages of the Experiment and Deploy part of these tools.

  • Second:

    • Have a good unit test coverage for the direct calls to OpenAI with VCR and reimplement some of the basic eval metrics that relevant to my context like Answer Relevance, Context Recall, or Context Relevancy.

    • Or use something like Promptfoo, Ragas or GuardRails for the local unit tests.

I haven't decided which product to use yet, but reading through their documentation, examining the code of their SDKs, and exploring their offerings has been a good learning process already.

In no particular order, here's the current list of evaluation products/projects that I'm aware of (although I haven't had a chance to thoroughly check all of them yet):

• https://mlflow.org

• https://www.vellum.ai/

• https://athina.ai

• https://www.patronus.ai

• https://wandb.ai/site/weave

• https://autochain.forethought.ai/

• https://ragas.io/

• https://github.com/agenta-ai/agenta

• https://klu.ai/

• https://www.braintrustdata.com/

• https://humanloop.com/

• https://www.parea.ai/

• https://docs.llamaindex.ai/en/stable/understanding/evaluating/evaluating/

• https://www.langchain.com/langsmith

• https://github.com/langfuse/langfuse

• https://www.rungalileo.io/

• https://www.guardrailsai.com/

• https://deepchecks.com

• https://arize.com

• https://github.com/UKGovernmentBEIS/inspect_ai

If you're using a tool that you really like or if you have any feedback or a different perspective on the market, I'd love to hear your thoughts.

While I haven't done much hands-on AI engineering this last two weeks, I've been keeping up with theory. Here's some of the notes:

Daily Prompting Paper

For a while now, I've been checking Arxiv daily for new papers on Prompt Engineering. It's a topic that's usually easy to understand, even in academic papers, so great way to kill time between tasks.

Many poor results in AI come from inadequate prompting skills + techniques like prompt chaining and retries, weren't practical before, but now make sense.

In June, this excellent paper was published. I spent the last couple of weeks reading it thoroughly, along with many of its citations. I've compiled some notes here, and I'm also planning to create a better structured resource with more examples and additional notes I didn't include.

Meetups and Conferences

I've been attending several meetups and conferences, including AIQCon yesterday. It's interesting to see the growing focus on AI Engineering (meaning implementing GenAI in products). Last year, there a lot of interest in LLMs themselves, but now the emphasis is on making LLMs perform well in real-world applications.

Key topics include:

• RAG (Retrieval-Augmented Generation): Techniques to improving embeddings, content augmentation, and advanced techniques like GraphRAG.

• Evaluation: Setting up good test suites, deciding what to test, and determining who should write the tests.

• No fine-tuning: The consensus is to focus on prompting, RAG, and evaluation before considering fine-tuning. Lot of pain and scar from fine-tuning already.

While some events feel like they have a high noise-to-signal ratio, I always learn a lot as a newcomer. Also, it’s such a great time to live in SF. For future conferences, I plan to research the participants and their topics more thoroughly beforehand.

A useful resource from yesterday's conference: https://huyenchip.com/llama-police.html by Chip Huyen (well worth following on X).

Books

On the topic of learning more theory, I'm currently reading two books:

1. Programming Machine Learning

https://www.amazon.com/Programming-Machine-Learning-Zero-Deep/dp/1680506609

I first read this when it was released, but I've forgotten most of it. Although it's a bit outdated and pre-LLM, I like the author's writting. Given how close AI Engineering is to traditional ML, I want to revisit this material.

2. Neural Nets mini-course by 3Blue1Brown

Another basic entry level material I was revisiting this week is this mini-course on neural nets: https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

2. Oreilly’s AI Engineering book

https://www.oreilly.com/library/view/ai-engineering/9781098166298/cover.html

This book was recommended by a close person, and I attended a talk by the author yesterday - it was one of my favorites, despite being too short. Only three chapters are available now, so I'll keep an eye out for future updates.

Context: In May I left my full-time job to start a 3-month LLM learning cycle by building a few products of my own. I’m documenting my journey here. Every week I post a recap of how the week went, things I learned, etc.

Had a bit of a shorter week. I went on a camping trip to Yosemite and had no service for 3 days. Despite this, still managed to pack 3 days of 10 hours.

Here's some of the progress and learnings:

Canopy AI: JSON vs XML and streaming

I'm still helping maintain the AI Assistant I built for Canopy. It has the usual Chat experience where a user asks a question, we do a RAG search and layer a bunch of things on top.

One frustrating aspect has been GPT-4's unreliability in returning JSON. OpenAI's Completion API has a JSON mode that ensures valid JSON output, but this option isn't available on the Assistant API when using the File Retrieval tool (their built-in RAG).

Although OpenAI's built-in RAG gives me better results compared to Pinecone and PG Vector, the trade-off for Canopy's use case was losing the JSON mode.

Canopy's assistant does more than just RAG the content. It runs categorization, sensitivity checks, confidence checks, and other tasks that might trigger subsequent LLM calls or be needed in the user interface. These constraints and extra information are metadata that I used to include in a JSON (making a single API call for speed and cost).

The problem with JSON is streaming. Ideally, you want to start rendering the message as you receive the response. In the JSON, I had a markdown answer as one of the values. I'd buffer the JSON until the right value starts streaming and relay that to the user interface, stopping when the value ends. A better solution would be to have the markdown first, followed by the metadata in a structured data format (JSON, XML, etc.). After a full day of prompt engineering, I couldn't consistently get GPT-4 to return the metadata in this format after the markdown message – until GPT-4o.

Last week, I spent a day migrating from the JSON format to a structure with markdown first, followed by the metadata inside XML. Anthropic uses XML in their examples, so I've been using this tip with OpenAI as well (makes sense if you think about it, it’s easier for transformers to handle words than curly braces).

The end format looks like this:

Markdown answer…
```xml
<metadata>
…
</metadata>
```

Incorporating the ``` is another trick that took me a while to learn. When you can't use JSON mode, OpenAI will often add wrappers like these. Instead of trying to remove them in the prompt or handling failures in the app, including "shots" (examples) with the ``` as part of the answer will guide the model to always return it.

This simplified the code on our app side and made the metadata correct nearly 100% of the time. The next step is to auto-retry on the rare cases when it fails, which I haven't implemented yet.

The New Product

So far, in these 3 weeks, I've spent around 40 hours on the new product, with a significant portion of that time dedicated to boilerplate tasks that I won't need to repeat for future products, which is a positive.

While most of the AI work was already done, I couldn't resist adding some extra features at the last minute with GPT-4o. Scope creep? Why not!

I finally have a name – but I’ll share it once the landing page is up. Until now I had it under a code name, but now I have the final name, domain, and a marketing landing page 90% done.

Selecting the domain and name included initial SEO work. I think it’s BS that domains are not relevant for SEO anymore. If you are a nobody in page authority, I think it’s a good idea to make the domains very literal, so that’s how I’m starting: a short name followed by something literal.

In the 2 days I had after spending a day on Canopy, I worked on:

• Feature work:

  • Refining the AI features and adding some extra parallel LLM calls.

  • Finished an Insights page (I hate styling Heatmaps).

  • Added dark mode support while stuck in traffic driving to Yosemite.

  • Started with a Telegram bot.

• Marketing

  • Finished a landing page and registered the domain.

  • SEO: Research to find keywords I could try to rank for and figure out a strategy for evergreen content (I'll need more time here).

Before sharing the product with everyone, I still have to polish some final details, like setting up the Stripe account, deploying to the final servers, etc.

Timeline-wise, I'm thinking I need:

• 1 more full week for polishing, details, and putting up the waitlist (including boring things like Terms of Service).

• About 3 days for the initial marketing work, including backlink work, evergreen SEO-focused pages, and some community-based distribution.

Then, I'm switching attention to the second product that I want for myself (while keeping 25% of my time on this one).

Once the product is live with users, I plan to share some of the lessons more openly by writing specific posts about certain things.

For example, on this project, I've been using a mix of multiple parallel LLM calls and OpenAI batch processing. These two things could be interesting as standalone posts.

I also hope to carve out time to write separate posts about some of the product and marketing aspects whenever I can.

Week 4 Plan

This week, I'll be splitting my work into three parts:

1 day for Canopy

2 days on New Product

3 days helping a friend with his startup (I usually work a 6-day week anyway)

I'm avoiding doing any kind of external work, but in this case, I made an exception.

It's a fixed-scope project on something I'm quite familiar with, plus it's an opportunity to learn some new React on a real product designed by great programmers. On top of that, what this friend is building is an impressive product in an interesting domain with a capable team, so I'll help whenever I can. I’m estimating probably 14 days on this (let’s see how off I’ll be this time).

This will delay my LLM learning cycle plan a bit, but it's a good trade-off.

I'll continue to share my updates here regardless.

Previously, people were using the inferior free GPT-3.5. With GPT-4o, there's no excuse for uninformed takes flooding social media and YouTube. OpenAI's terrible ChatGPT UI and closed-source GPT store are still problematic. At least Anthropic is trying to teach better prompt writing (with more effort than OpenAI).

But back to my rant...

Many users do not try to tune the system, ask poorly written questions, get unsatisfactory results, and then complain online. We're 2 years into this cycle - it's time to wise up.

By now, we should all know that these systems are few-shot predictive models that respect the chain of command. Yet users provide neither the few-shot examples nor the command it should follow. It's no wonder ChatGPT-written resumes and landing pages sound like professional BS with poor English.

To get good results, you need to:

1. Give it examples

2. Understand the chain of command

GPTs can handle Zero-shot and Few-shot (but not really)

• Zero-shot learning: performing new tasks using broad pre-training

• One-shot and few-shot learning: generalizing from 1-to-many examples to perform a task on new inputs

While GPT should handle both, its zero-shot capabilities aren't there yet beyond toying around. Always provide 2-10 examples of what you want - otherwise, you'll get predictions based on general knowledge, which means bad results.

The GPT Chain of Command

1. OpenAI and other providers set the basic rules and capabilities.

2. Developers can customize these settings for specific tasks.

3. End-user requests go through the customized settings.

For non-developers, the best way to leverage the chain of command is to create a custom GPT with a system message specifying your requirements, along with necessary examples.

If you're coding against the APIs directly, you have more options beyond just system messages, such as fine-tuning. However, in the context of ChatGPT, system messages are the primary tool available (function calls and tags can be discussed in another post).

Tip: You can use Anthropic's prompt generator to create the initial system message and examples, then paste it into ChatGPT's custom GPT (Anthropic's UI is too bad to use directly): https://x.com/danielvlopes/status/1792978707201831155

Here's a simplified version of what I use for my Blog Posts custom GPT in ChatGPT:

# Identity 
You are a draft polisher assistant. Your job is to receive long-form poorly formatted drafts and convert them to a final version maintaining the tone, and vocabulary but using fewer words, fixing the grammar, adding more spacing, better structure, and making things clearer.

# Steps to take

To do this, first carefully read through the entire blog post to understand its structure and key points. Then, go through and rewrite the polished version of each section while keeping the same titles and technical insights.

Here are some tips:

- Remove unnecessarily complex language. Replace these with plain, straightforward terms. For example:
-- Instead of "leverage", use "use"  
-- Instead of "synergize", explain what you actually mean, like "work together"
-- Instead of "disintermediation", just explain the concept in simple terms
- Break up long, run-on sentences into shorter, clearer ones
- Use contractions like "it's" and "you'll" to make it sound more natural and conversational 
- Identify the technical parts of the text and don't remove these
- Keep the tone straightforward and professional, avoiding overly casual language but ensuring it doesn't become too formal.
- Keep the personal touch and original intent of the content.
- Format the condensed version of the post with clear section breaks and spacing for readability.

Here's a list of banned words to never ever use any of the following words (or similar words): "delve", "embarked", "journey", "thrilled", "aim", "tailored", "fledgling", "shrouded", "endeavor" , "quantifiable", "empirical", "whereabouts", "debut", "keen", "craft", "tailor", "elevate", "ignite", "empower", "unleash", "horizon", "harness", "diving into", "interconnect", "enhance".

As you write the new version, make sure to maintain the same voice, tone, vocabulary, and style as the original blog post. The goal is not to rewrite it completely, but to have a slight more polished version that is easily digestible format for people short on time.

# Example 1

Input: 
```
Original draft
```

Output: 
```
Well written version
```

# Example 2:

Input: 
```
Original draft
```

Output: 
```
Well written version
```

# Example 3:

Input: 
```
Original draft
```

Output: 
```
Well written version
```

Put in some effort; otherwise, you are to blame for the bad results. With early-stage technology, we can't have the luxury of expecting things to work perfectly right out of the box.

Idea selection

As I said before, I'm currently focused on small indie-hacker-scoped projects. What does that mean?

I'm sorting ideas by level of scope and potential size. For the first projects, I'm avoiding super ambitious things, they need large surface area and tend to have a lot of competition. In the middle, there are “during gold rush, sell shovels” type of ideas, like AI infrastructure tools for developers where the need is clear, but so many teams are rushing to build them.

I'm looking for small things or those with seemingly bad business models that I want to use myself. When picking a market, you either solve something for yourself or others. For now, I'm choosing things I want for myself to save time on initial customer research.

However, I still want something with clusters of people easily found online so marketing isn’t an uphill battle.

To recap: I need something not hard to market with a bad TAM, small in scope, and that I want for myself (likely small apps, wrappers, or add-ons for existing players).

Perfect recipe for disaster 😅 Why?

1. It gives me time to focus on adjacent things I'll reuse later, like a clean boilerplate with UI libraries, authentication, billing, and API clients.

2. In this first month, I'm basically starting easy to rest a bit while staying creative, studying, and even having a chance to build some baby MRR (but very unlikely).

I've narrowed my list down to 5 potential ideas for now, and I’m already working on a side project that I’ve had started in the past. These ideas will be for the next project.

Week recap: progress!

Boilerplate

Rails 🚂

I'm sticking with Rails - it's stable, battle-tested, and I know it well. Not interested in Next.js yet, maybe just for marketing later.

Going with the vanilla Rails stack for now, using Hotwire instead of React for the front end. Everything has a learning curve, and modern-day React is one I don't want to tackle now. Might use React later for more interactive ideas.

Jumpstart Pro 🚀

Didn't want to waste time setting up billing (even with Stripe), accounts, authentication, 2FA, or even little stuff like pagination. Went with Jumpstart Pro - it's a good starting point with high-quality code that's easy to customize, even if I'd prefer a smaller and with less dependencies. JSP also has a lot of new things that I wasn’t familiar with like Dev Containers, so fun learning experience.

TailwindCSS + TailwindUI (Pros & Cons) 🌬️

Switched from theming Bootstrap (like I did for Canopy) to TailwindUI. Not having CSS files makes it perfect for code generators like custom ChatGPT and V0, which are great at generating Tailwind markup. TailwindUI is also very versatile and covers a lot, so I get a productive boost from the (copy, paste, and customize).

However, using TailwindUI with vanilla Rails hasn't been smooth. I find myself coding basic things because TailwindUI has no vanilla JS option for behaviors like dropdowns, carousels, and off-canvas panels. Jumpstart's Stimulus helps with some of that. Another issue is that TailwindUI's markup was meant for React components, so you often need to abstract classes into Rails helpers for repetitive use.

This week: 2 days spent to cover TailwindUI basics, 1 day to learn Jumpstart, and the learning curve of TailwindCSS every day. But now feeling quite productive already.

Brainstorming with friends

I have a lot of people reaching out to network and brainstorm ideas. Last week I spent about & hours on this.

Canopy: GPT-4o

I work 8 hours a week at Canopy, doing 1-2 hours at a time. After OpenAI launched GPT-4o, I migrated our AI Assistant:

• It's much cheaper and faster (so fast that streaming isn’t necessary anymore)

• Overall better results: I tested using our rubric of 200 questions (not a formal eval system yet). After migration, the results were mostly better.

Two small issues:

1. Acronyms: It made up some acronyms, which I fixed with prompt engineering.

2. JSON: We use OpenAI's Assistant API with their RAG tool, so we can't use "JSON MODE". I ask for JSON via prompt engineering. It tried to add a wrapper around the markdown to avoid breaking the JSON, which we fixed by adjusting the code.

The speed and price drop let us split our 200+ line prompt into two prompts, running the query sequentially:

1. Ask for the answer, present it to the user

2. Ask for additional metadata async and refresh that part of the screen (WIP, shipping next week)

The cost drop opens up the opportunity to use GPT-4 level of IQ for everything, including B2C customers with low subscription prices.

Product 1: A Lot of Work Done

As I said before, I'm already working on my first product, which I've been building since early this year a few hours a week. It's a small thing, but a useful tool for myself that benefits from LLM support.

I spent 3 days (30 hours) migrating the code to Jumpstart Pro and 4o, and adding features for launch - shooting for next weekend. I also worked on the marketing landing page.

Still to do: Complete an “insights” feature. Add an LLM-powered overview screen. TailwindUI is helping a lot with the design. So, I’m estimating ~3 to 5 more days before able to onboard users.

I’ll need to setup new servers due to significant changes from Jumpstart Pro. Setting up servers again will take likely at most one day.

Goal: For this week or next, put the product up + add a marketing landing with a waitlist, and send it to the communities I’m a member of.

The bad things

1. Poor sleep hygiene: Worked too close to bed, messing up my sleep. Getting older I need to be a lot smarter about sleep hygiene these days or the price is real.

2. Not enough exercise: Only exercised 3 times: 2x 40min half-assed weight sessions, 45min hard VO2 max virtual cycling race on Zwift. Usually do 2x that, it's important for my mental health.

3. Missed meet-ups: Trying to go to more. Had two interesting ones this week, an Agent Guidance and a Gemini vs 4o Hackathon on Saturday, and skipped both.

Next week

I'm off for 3 days on a last-minute camping trip, so this week will be short. Working this weekend to account for it. Most of my time will be split between working on Product 1 and some LLM work for Canopy.

I've been toying with LLMs since end of 2022, trying to create an AI Assistant for my company, Canopy. The tech wasn’t quite there until GPT-4 came along. Last week, after launching our AI Assistant in alpha, I decided to leave my job to focus on developing my own ideas. I’ll be documenting my journey here as I go, and this is the recap of my first week.

Why now?

With the last couple of years' progress, so many of the traditional ML projects have been (or could be) replaced by language models, making things easier compared to just three years ago. Things like unstructured data, classification, sentiment analysis, NLP, speech, and much more are now all manageable by one person or a tiny team.

I think there are four ways to benefit from what is happening:

1. Big Tech working on foundational infrastructure or large problems.

2. Moonshot projects challenging existing players like Perplexity vs Google.

3. Vertical applications (or what people are now calling "Applied AI").

4. Indie hackers focused on things that don't look venture-scale at first glance.

For now, for these next three months, I'm personally interested in item 4. Maybe later I'll try to join a team working on the other ones.

Setting a 3-month deadline

Having a short deadline helps keep me motivated and accountable. It pushes me to focus on practical, achievable goals.

Since I work in product and not ML or research, I'm interested in end-use application. There's a ton of effort involved in making LLMs (unstable, unreliable & expensive tech) work smoothly in a finished product.

For instance, if I were only studying, factors like accuracy, cost, or user experience would be glossed over. But if I need to ship and charge for something, that’s the opposite.

Three months, with my skill set, I think is a doable but still challenging timeline to ship at least two products (my stretch goal is six).

Now, to the first-week recap

I had 5 goals for this past week:

1. Ship my personal site, start a Substack, clean up social media.

2. Get a boilerplate codebase for my projects (sign-up, subscription, etc).

3. Organize and research my ideas: pick 3 to 6.

4. Canopy: monitor the Alpha launch of our AI Assistant.

5. Bonus: Ship a WIP project that I had already.

✅ Goal 1: Personal site, Substack & social media

Personal site: Over the weekend, I started working on a personal site that I’d need to send to some people. I also used it to play with Next, Vite, Astro, Tailwind, and Shadcn. All things not part of Canopy stack.

I went with Next + TailwindUI. Site is live here: https://daniellopes.dev

For my products will be using just Tailwind and my usual Rails.

Substack: Working with the garage door open is something I always wanted to do for Canopy, but I left that for my co-founder. So, last week I started this, and I posted the first post: Using LLMs to parse 7 years of stand-up updates (which was an exercise in trying different summarization approaches as well).

Writing a ton is also part of one of my ideas that I might work on next month.

Social media: To stay more informed about things happening in San Francisco (meetups, hackathons, etc). So spent a few hours making list of people I want to follow more closely. Social media isn't all just noise; it can be valuable.

🟨 Goal 2: Organize all my ideas

I made a lot of progress on this goal, but not enough to confidently have three solid ones. I'll keep working on this next week. Here's the gist:

I keep a list of ideas, logging anything that comes to mind (features or products). I always have anywhere around 100 to 200 items.

This week, primary goal was to organize my list, considering these 4 criteria:

• Is it a newly viable thing? And is it flooded with people doing good work already? (Areas like Knowledge Base for Companies or Customer Support are already saturated.)
  

• Do I want this for myself? Building something for others demands a lot of research, which isn't feasible within a three-month timeline.
  

• Does it have commercial potential?

  • If no commercial potential, the aim should be for learning or personal use; don't waste time on unviable or tarpit ideas. Do just enough to meet the goal and move on. At the end of the day, I want people to use the things I build.

  • If there's commercial potential, do I have a unique perspective on solving the challenges? Then do the usual analysis—pricing, customer base needed for a MRR, competitors, etc.
    

• Does it have the potential to find a single feature that could be a lot better than existing methods? And can it be developed as an add-on for existing players or does it need to be a replacement? (replacements are less interesting with this timeline).
  

• What’s the marketing effort? Can I market it alone? If it's B2B, can I identify a sustainable group of people to target? If it's B2C, can I come up with marketing strategies that don't rely on ads or require excessive time?

Many ideas I had on the list from last year had no credible competitors six months ago, but now have promising ones. So, I ditched those. While I don't mind competition when working with more resources, working solo, I don’t need my back against the wall all the time.

✅ Goal 3: Boilerplate

Initially, I planned to use much of Canopy's codebase for non-proprietary elements and create a standard boilerplate things like sign-up, subscriptions, and admin (which would let me contribute back improvements).

However, given my time constraints, I'm reluctant to spend excessive time cleaning things like our complex billing system that covers B2B, B2C, and enterprise solutions.

I've decided to go with Jumpstart Pro. There are several things I plan to remove/change, but for now, it's enough. I’ve also wanted to learn Tailwind for a while, so it was nice that Jumpstart Pro came with Tailwind already.

I’m also sticking to Rails (The One Person Framework ™️).

I’ve been tempted to make the jump to Python because of how readily available tooling is for their ecosystem, but the fastest way to ship nothing is to pick a bunch of things you are not expert on. It’s too easy to underestimate all the ecosystem details that you need to know in order to ship a product that is stable enough to require nearly no-maintenance.

In this case, my goal is to focus on the behavior of LLMs (different models, prompt architecture, cost, reliability, etc.) and not mess around with different frontend or backend technologies.

✅ Goal 4: Canopy - Monitor the Alpha of AI Assistant

We launched our AI Assistant in Alpha last week. So, I focused on getting visibility for how the system is performing. We haven't fully committed to an evaluation strategy yet. For now, we managed to see how satisfied people are, along with some basic analytics and reporting on content that we didn't have for RAG.

For “eval” this looks very promising: https://athina.ai

I’ve seen their product at a meet-up in SF last week and found it a better fit for our needs than other tools I'd considered (open source solutions like OpenAI’s or Ragas.)

❌ Goal 5: Ship a WIP product

I didn't succeed with this one.

Last year, I started working on a product that solves a personal use. None of the business calculations add up (cost per person vs what I can charge, TAM, competitors, etc.), but it meets my criteria of being something I want to use and something I can learn from.

I've already learned a lot from this project. It's what I worked on over the weekends last year to learn different models, prompting or RAG architecture without getting burnt out from just working at Canopy. It's ~80% done, but now I just need the last stretch to make it possible for others to sign up & buy it so I'm not paying for other people's API calls.

I'll probably need another three days to get it ready for other users. Since I chose Jumpstart Pro, I'm going to migrate the product to it and see how that goes. So, this will carry over into Week 2.

Plan for Week 2

Switching jobs, picking ideas, setting up the basics, and wrapping up existing projects is kind of like tying up loose ends. It's not the most thrilling, but it's still important.

For next week, I have 3 main goals: Launch (1) and market (2) the project I just mentioned, and to do more research on my ideas (3).

Custom GPTs is the feature from ChatGPT that makes me use it more than anything else (more than Claude, Perplexity, or Gemini). I have a handful Custom GPTs that I use every day, and I create a bunch for specific needs as well.

One of my favorite use cases for one of my recurring Custom GPTs is to brainstorm different solutions before I start coding (this one is using a Rails-specific GPT one). I pretty much will start planning my work there. It will often give me good suggestions or at least provide me with some good naming for things. Then at the end a finish by asking for a review as well.

In this example, I know what different patterns I might want to use here. I could do it myself no problem, but I find it great to be able to see a draft version of all the different ideas in 5 seconds first and then decide whether to try to do it in my head, picking just one and spending 20 minutes executing on it.

The request

I need to reuse these lines on a different controller. What would you suggest? Maybe should I extract a service or PORO class? or a concern for controllers?

@learning_path = current_person.personal_learning_path
@learning_path_modules = @learning_path.training_modules.includes(:lessons) if @learning_path
@learning_path_progress = @learning_path.progress_for(current_person) if @learning_path
@training_modules = load_training_modules

---

def load_training_modules
  if current_organization_training_subscription&.trial?
    TrainingModule.standard_order_for_trial
  else
    TrainingModule.preferred_order_for(
      current_person,
      order_by_preference: "recommendations",
      exclude_seen_modules: false,
      consider_watched_lessons: true,
      alternate_results: false
    )
  end
end

====HERES THE FULL CONTROLLER=====

class TrainingModulesController < TrainingBaseController
  menu_item :training_modules
  before_action :initial_breadcrumbs

  def index
    @assessment = current_person.last_assessment_with_self_response_complete
    current_person.generate_personal_learning_path_if_needed!

    @learning_path = current_person.personal_learning_path
    @learning_path_modules = @learning_path.training_modules.includes(:lessons) if @learning_path
    @learning_path_progress = @learning_path.progress_for(current_person) if @learning_path

    @training_modules = load_training_modules
    @display_tour = !current_navigation_preference.learning_paths_tour_closed?
  end

  def show
    @training_module = TrainingModule.includes(:lessons).find(params[:id])
    @progress = TrainingModuleProgress.new(current_person, @training_module)
    @lesson = @progress.next_available_lesson || @training_module.lessons.first

    current_person.training_history_entries.track(request.url, @training_module)

    redirect_to training_module_lesson_path(@training_module, @lesson)
  end

  private

  def initial_breadcrumbs
    breadcrumb.add "Modules", training_modules_path
  end

  def load_training_modules
    if current_organization_training_subscription&.trial?
      TrainingModule.standard_order_for_trial
    else
      TrainingModule.preferred_order_for(
        current_person,
        order_by_preference: "recommendations",
        exclude_seen_modules: false,
        consider_watched_lessons: true,
        alternate_results: false
      )
    end
  end
end

GPT Response

Given the repetitive use of the code involved in fetching and preparing `@learning_path` and related entities across different controller actions, extracting these operations into a reusable component is a good idea. This will help maintain DRY (Don't Repeat Yourself) principles and ensure cleaner and more maintainable code. Here are a few options to consider:

### 1. Service Object

A service object is ideal for encapsulating business logic that doesn't naturally fit within a model or a controller. Since your code involves preparing data for views based on user-specific and contextual business rules, a service object could be a good fit.

# app/services/learning_path_service.rb
class LearningPathService
  def initialize(person)
    @person = person
  end

  def fetch_data
    learning_path = @person.personal_learning_path
    {
      learning_path: learning_path,
      learning_path_modules: learning_path&.training_modules&.includes(:lessons),
      learning_path_progress: learning_path&.progress_for(@person),
      training_modules: load_training_modules
    }
  end

  private

  def load_training_modules
    if @person.current_organization_training_subscription&.trial?
      TrainingModule.standard_order_for_trial
    else
      TrainingModule.preferred_order_for(
        @person,
        order_by_preference: "recommendations",
        exclude_seen_modules: false,
        consider_watched_lessons: true,
        alternate_results: false
      )
    end
  end
end

Usage in a controller:

def index
  service = LearningPathService.new(current_person)
  data = service.fetch_data
  @learning_path, @learning_path_modules, @learning_path_progress, @training_modules = data.values_at(:learning_path, :learning_path_modules, :learning_path_progress, :training_modules)
  @display_tour = !current_navigation_preference.learning_paths_tour_closed?
end

### 2. Concern

If the logic is tightly coupled with ActiveRecord objects or used across multiple controllers, creating a concern might be a better approach. This keeps controller-specific logic within the context of controllers.

# app/controllers/concerns/learning_path_loadable.rb

module LearningPathLoadable
  extend ActiveSupport::Concern

  def load_learning_path_data
    @learning_path = current_person.personal_learning_path
    @learning_path_modules = @learning_path&.training_modules&.includes(:lessons)
    @learning_path_progress = @learning_path&.progress_for(current_person)
    @training_modules = load_training_modules
  end

  private

  def load_training_modules
    if current_organization_training_subscription&.trial?
      TrainingModule.standard_order_for_trial
    else
      TrainingModule.preferred_order_for(
        current_person,
        order_by_preference: "recommendations",
        exclude_seen_modules: false,
        consider_watched_lessons: true,
        alternate_results: false
      )
    end
  end
end

Usage in a controller:

class TrainingModulesController < TrainingBaseController
  include LearningPathLoadable

  def index
    load_learning_path_data
    @display_tour = !current_navigation_preference.learning_paths_tour_closed?
  end
end

### Decision

The choice between a service object and a concern depends on how closely the logic is tied to the controller's responsibilities. If the logic is more about data manipulation and less about HTTP-specific operations, a service object is more suitable. If it's about setting up controller instance variables and integrates closely with controller lifecycle (like before actions), a concern might work better. In your case, either approach would streamline your code and enhance maintainability, with a slight preference for a service object given the nature of the logic involved.

Final decision

I ended up with the concern approach with a code that is very similar to the original GPT suggestion.

Custom GPT “source code”

This custom GPT is the one I use for Canopy setup for the following use cases (and a has few examples of our codebase inside):

I've been looking back into my daily stand-up updates and turning things into monthly or quarterly reflections under the #journal tag here. This note is from my first quarter at Canopy after doing some parsing and summarization over the last 7 years.

The company was bootstrapped and didn't have a technical co-founder yet. My partner had done a great job so far with the resources she had, but now there was some foundational work to do if we wanted to see some growth.

Month 1

In my first week, I set up Segment to integrate our marketing and sales tools like CRM and Drip Campaigns, and to lay the groundwork for a more robust analytics system.

Our in-app analytics was slow and crashing. So it had been turned off and we had no reports for things like engagement or revenue per account. Sales had somewhat long cycles driven by contracts with a high upfront cost; we had a pay once per email address pricing at the time that was likely masking the real MRR/ARR.

During the first month, my goal was to improve admin performance and get our reports running again. I did this without setting up a new BI tool, although connecting Segment to a warehouse using BI tools like Hex or Mode would have been smarter.

I also streamlined our sales process by integrating Freshsales, added an Intercom chat for user feedback, and upgraded to Rails 5.

My main focuses in the first 30 days were:

1. Lowering the error rate.

2. Updating our tech stack.

3. Implementing key marketing, sales, and support tools.

4. Getting visibility on key metrics like retention, engagement, and LTV.

Month 2

We launched a Knowledge Center feature within the product—a collection of eBooks that users can read directly in the app. This required building a simple CMS with markdown support and PDF download functionality. Interestingly, this was the initial feature that would validate the interest in content as a service, which we fully embraced in 2023.

I also began building our Slack integration. At the time, our system relied heavily on email, and most customers had migrated to Slack. Investing in Slack integration proved to be a good decision for engagement.

Additionally, I helped with sales enablement initiatives, like deploying a read-only demo environment. I also supported some marketing efforts like a hosting platform for an email-based mini-course.

Month 3 and 4

We shipped our Slack integration and improved our billing system to enable future recurring pricing tests. We also fixed several billing issues.

It became clear that engagement was strong in the first months but tapered off (it made sense for the feature we had). I surveyed customers and analyzed user session recordings to understand this better.

We realized our product wasn't ready for a dedicated sales role yet. It was time for founding sales. To achieve better product-market fit, we unfortunately had to let go of our sales person, and my co-founder took over sales responsibilities to enable founder-led sales.

--

Reflection

In my opinion, product-market fit (PMF) works in levels. What we had was more than sufficient for a great small business of 2 making up to $200k each, but it wasn't enough for our SaaS compound growth ambitions. We didn't have PMF for the next level. And trying to find PMF, you don’t need things like sales quotas or bonuses interfering with the process.

Also, get visibility as soon as you start getting some volume. When you are on low hundreds of users, you may not need full-fledged BI yet. Tools like Smartlook or a basic activity feed per account might be enough. However, nowadays, I would add BI much sooner using Hex because it's so easy to set up and maintain.

I think everyone should have a Year Review doc and take note quarterly of all your work (projects, impact, launches, metrics, etc.) so you can use this in interviews and for your resume.

This is a problem that I have myself. I've spent 7 years at Canopy, wrote most of the 200kloc of our codebase, helped a lot with our UI design, was involved in all product and hiring decisions, helped fundraise, etc.

But my LinkedIn is empty. I honestly don't even remember what I did.

The thing is that for 7 years I've used Canopy Perform to record daily standups. I have a journal of everything. Now, how do I make sense of it?

Summarization

So I decided to summarize everything with GPT4 and Claude Opus and see the differences. The steps:

1. Exported the CSV with all my Heartbeats from Canopy Perform. The data was a row for every day containing HTML with lists, strong tags, and divs of what I did on that day, what I did the day before, and sometimes reflections and challenges.

2. Used ChatGPT with Code Interpreter to rewrite the CSV, combining every day into a new CSV that had weeks instead of days, and the value of each week was all the days of that week combined.

3. Wrote this script to call OpenAI gpt-4-turbo-preview and summarize every week while getting rid of the HTML (tried with GPT 3.5, and as usual, it's useless). The output wasn't perfect – I should have added more guidance on vocabulary to cut the BS words, but good enough.

4. Then I wanted to manually go over every month and try to reflect/remember what I did. For that, I decided to test the differences between a Custom GPT in ChatGPT, OpenAI Playground, and Claude Opus with the same system message.

Comparing Claude, Custom GPT and OpenAI API

Here's the comparison of each on my 1st full month at Canopy:

This was the System Message (same used for all), and here’s the input data.

Note: the summary was generated by GPT4 with low guidance, so there's some BS vocabulary that I would never use.

Claude console result

All default settings: temperature 0, claude-3-opus-2024, max tokens sample 1k

Title: April 2017: Focusing on Admin Panel, Rails 5 Update, and Slack Integration:

In April, I focused on improving our Admin panel, particularly the statistics section, which was deactivated due to performance issues. I also worked on updating our system to Rails 5 to keep pace with upcoming updates and facilitate a smoother transition to Rails 5.1 in the future. 

Additionally, I began integrating Slack into our workflow to enhance our communication and collaboration processes. Although I had to postpone some administrative tasks, such as completing revenue and question popularity reports, I prioritized these system updates and integrations to ensure our technology remains current and efficient.

ChatGPT with a Custom GPT:

**April 2017: System Enhancements and Administrative Focus**

This month involved significant focus on improving our Admin panel, particularly the statistics section, due to performance issues. I managed the transition from CRM and Intercom management to addressing these concerns, aiming to restore and optimize this vital tool for our team’s decision-making.

Following this, I tackled pending administrative tasks and began updating our system to Rails 5, balancing these priorities effectively. The update was crucial for keeping pace with new technology and was strategically planned to ensure a smooth transition to future versions.

I also initiated integration with Slack, enhancing our workflow efficiency, although this task is ongoing. Overall, the month was about enhancing system performance and efficiency while managing routine administrative duties.

OpenAI Playground (GPT-4-turbo-preview)

Default temperature 0.5 with gpt-4-turbo but max tokens increased to 1k

### Monthly Work Summary: April 2017

This month, I focused on various critical areas in our system administration. I began by addressing performance issues in the statistics section of our Admin panel, aiming to enhance its efficiency and restore its functionality.

Following that, I handled pending administrative tasks and initiated the upgrade of our system to Rails 5. This update was important to keep our system compatible with upcoming versions and improve overall performance.

Additionally, I planned to integrate Slack into our workflow to streamline communications. Each of these tasks was vital for maintaining and improving our operational capabilities.

I think my preference order here is:

1. Claude Opus

2. OpenAI Playground with GPT-4-turbo

3. ChatGPT with Custom GPT

What I actually did

Title: April 2017: Focusing on Admin Panel, Rails 5 Update, and Slack Integration

In April, I focused on improving our Admin panel, particularly the statistics section, which was turned off due to performance issues. 

I also updated us to Rails 5 to keep facilitate a smoother transition to Rails 5.1 in the future. Additionally, I began the work on the Slack integration feature. Slack took priority over some of the extra Admin things I wanted to do like adding a Revenue and Engagement reports.

Some of these things got lost in the first pass of summarization that GPT-4 did on the Python script above.

Conclusion

After running through all my ~88 months at KYT/Canopy, I think this was a very helpful reflection exercise to see all the things I did. It helped me reflect on my mistakes and on things I did right.

Moving forward, I plan to keep a monthly journal like this and keep it up-to-date here on Substack under the Journal tag. I’ll share a post a day in retrospect, covering all these last years. Here’s the first one:

I've celebrated multiple achievements at work. We rebranded KYT to Canopy in just three weeks, launched our iOS app, and shipped our AI Assistant. But, last week was also my last full-time at Canopy.

Things weren't always easy. Going after the ambitious goal of helping people become better managers at scale with a bootstrapped small team required a lot of effort, and I am so proud of what we accomplished.

When I joined, for a long time, it was just Claire and me. We grew it into a business making 7-figures in annual revenue, an amazing team of 8 people, and reached tens of thousands of users. Working with Claire and the team has been amazing.

But after seven years, it’s time for something new. So I’m moving into a technical advisor role at Canopy.

I’m not exactly sure what comes next. One thing I am sure about is that I want to work with LLMs a bit more.

My last project at Canopy was our AI assistant. It was a blast to work on, and I learned a lot. This project is something I've been working on and off since GPT-3, and now it’s up and running beautifully.

There's so much potential. For now, I’ll focus on learning more about similar applications. I learn best by doing and I have several ideas involving LLMs—some more personal, others with maybe business potential.

I’ve set aside three months to work on these projects. If I like the direction, I might try to extend for another 3 and see from there.

I'm going to share my experiences on Substack / Twitter / LinkedIn.

This is the first time since moving to the US that I have changed roles without another job lined up. It feels weird but also exciting.