Apple history, behind the scenes: 50 stories from the people who built the Mac, NeXT, iPod and iPhone

Fifty stories from the CHM oral histories: how they got stuck, who said no, and what it cost to ship.

I recently published two pieces on Apple's second act: a timeline and a guide to the interviews.

While writing them I kept cutting the same kind of thing: an engineer or a marketer going deep on one problem. How they got stuck, who said no, what it cost to get unstuck. It was the best material and it didn't fit.

So I put it all here.

Stories like the one Andy Grignon tells about running the radio team on the first iPhone. The phone had two processors, one for the apps and one for the radio. The wire between them would randomly die, the radio would reboot, and your call would drop. Thirty engineers looked for the cause and nobody found it. They shipped anyway.

This is a giant post, so I organized it in chapters, each with a tag: Management, Leadership, Engineering, Design, Product, Marketing. If you only care about one of those, Cmd+F it and jump around.

The performance review delivered on a walk, so there would be no copy

Management · The Macintosh team, 1982 to 1984

Andy Hertzfeld wrote most of the Macintosh Toolbox: the code in ROM that gave every Mac application its windows, menus, fonts and dialogs. He did it under a manager he could not work with, and the way that conflict was handled is why Apple lost him within weeks of the Mac shipping.

Two people need introducing. Bud Tribble was the Mac's first software manager, a medical student on leave who had come down to work on the project because Jef Raskin told him it would take a year. Burrell Smith designed the Mac's hardware. Together with Hertzfeld they were the core of a team that ran on the opposite of a chain of command: whoever had the problem talked to whoever could solve it, and Steve Jobs talked to everyone.

Tribble left in December 1981, a year and a half in, because his MD/PhD programme was about to dismiss him. Hertzfeld describes what he was losing.

I need a manager not to tell me what to do, but help me do it. And so, Bud was just tremendous. Also, I didn't have the confidence, really, that all these low-level decisions we had to make, I needed someone to corroborate me. So, Bud and I did all that together. (Hertzfeld, CHM oral history part 1, p.26)

He and Smith went to Jobs before the replacement was even hired.

I was worried that I was going to get a bad manager. Burrell and I both were, and I remember we both went and talked with Steve as soon as [...] we found out Bud was leaving, said, "We're not sure we can do this without Bud." And Steve made a promise to me that if I ever got a bad manager like that, he would defend me from it, which, we did get the bad manager, and he didn't defend me from. So, that was, that's essentially why I left Apple. (p.26)

The replacement was hard to find. The Mac wanted people at the edge of computer science and offered them 64K of memory and no memory management unit, because the machine was supposed to sell for $1,500. Several candidates turned it down. Bob Belleville came from Xerox PARC and, in his spare time, was building his own Xerox-style machine out of Intel chips; Hertzfeld took that as the sign of someone who liked to program. The two got along for about a year.

What broke it was a routine.

Steve would, almost every work day, come by around 6 pm, to see what was new, but Bob couldn't stay for that. He had a family and everything, and he hated us hearing about decisions from Steve before he did, or us telling Steve things, what issues [were] going on before we told them to Bob. And so, part of why he ended up just giving me a bad review for the period of time I wrote most of the Mac Toolbox, and he did it because he was upset about me talking with Steve. At least, that's one part of it. The other part of it is [...] I'm not easy to manage, I'm too self-determined. (p.28)

On Belleville's side of it:

He had worked in the Navy or something, or the Army. And he believed in chain of command. And, you know, things have to flow through the organizational org chart. It's the opposite of what the Mac was. [...] It was chaotic, you know, the way we were doing it. There's some legitimate criticisms, but he certainly didn't go about it the right way. (p.28)

Before the review there was a test. Belleville fired Bruce Horn, the young engineer writing the Finder, and Hertzfeld went to Jobs to reverse it.

He attacked Bruce Horn before he attacked me [...] because Bruce was an easier target than me. So, just to test the waters, as much as anything, he actually fired Bruce. I had to go to Steve and say, "He can't fire Bruce. You know, he's crucial, and we needed him." And Bob was very upset at me for doing that. (p.28)

Then the review itself, covering the stretch in which Hertzfeld wrote most of the Toolbox.

I was really shocked at the bad review, you know, because I was doing the best work of my life, and the bad review kind of said, "Your technical work is perfectly adequate, but your personality has to change." (p.28)

Belleville reported to Jobs, so the written review had to go through Jobs first.

He wrote a written review, that to this day I'd never seen, because Steve was his boss, and [he] had to run it through Steve, and Steve says, "You can't send this review to Andy. He'll freak out." So, what he did was, he procrastinated for a long time, but my review was way overdue. He decided to give it to me verbally, so there's no evidence of it or whatever. So I took a long walk, like 45-minute walk with him, where he gave me all this information. I started crying. I came back to Apple. He had to leave to go tend to his family. People saw I was crying, and [asked] what's wrong? I said, Bob just freaked me out. (p.29)

Steve Capps, another Mac engineer, went to Jobs right away, as Hertzfeld remembers it. Jobs set a meeting for the next morning.

He denied saying most of the things he told me, which flabbergasted me. How could you do that? You know, the problem could never be healed if you're not willing to admit that you even did it. And then [...] I think he was as scared of me as I was scared of him, and we just avoided each other. I had to do some emergency, crucial programming work at the time. But we just, he was out of the loop, basically, for that. (p.29)

Hertzfeld went on leave after the Mac shipped in January 1984, kept coming in every week or two, and eventually quit. On what would have kept him:

They definitely wanted me to come back, especially Steve, but [...] what they really should have done, and it sounds egotistical to say this, but he should have made me an Apple Fellow, where I would not have had to work for Belleville. It would have solved the problem. I couldn't ask for it, because it sounded like you have to be awarded it. You can't ask for it. But it was obvious to me, that I would have stayed. (p.42)

The precedent:

When they invented Apple Fellows, they made two Apple Fellows, Steve Wozniak and Rod Holt. And then, when Lisa shipped a couple years later, in beginning of '83, they made two Apple fellows, Bill Atkinson and Rich Page. When the Macintosh shipped, Burrell and Andy would have been good candidates if you were doing a similar thing, but for whatever reason, that didn't happen. (p.42)

Years later, at a Mac reunion, the two men apologised to each other. Hertzfeld's assessment: "I don't think it was super sincere on either of our parts." (p.29)

Selling Apple a three-times-faster graphics library for exactly what one manager could sign

Management · Outside Apple, looking in, 1986 to 1988

After he left Apple, Andy Hertzfeld kept building things for the Macintosh. Twice he sold Apple software it needed. The first sale went badly, and the second one worked only because of what he had learned from the first, plus a question about a manager's signing authority.

The first was Servant, a program that let several applications stay open at once, a generation ahead of what the Mac shipped with. Apple paid him $150,000 for it and never shipped it.

I sold it to Apple for $150,000, but that prevented them from... people were jealous I was getting paid or whatever, and it just wasn't the right thing. It had to have been done inside Apple, I later figured that out. So it was just a blunder. Made a promise to myself that the next thing, it has to make sense for me to do something, as a third party, has to make sense as a third party to do something for Apple, has to make sense internal at Apple. So I swore to myself that I never would do that again. I'd never do a thing outside of Apple that should be for Apple. (Hertzfeld, CHM oral history part 1, p.42)

He broke the promise almost immediately, and this time it worked.

The setting is Radius, the startup Burrell Smith founded to make large monitors for the Mac, where Hertzfeld held about ten percent as a founding shareholder and dropped in every week or two. In 1987 Apple shipped the Macintosh II, its first colour machine. Colour meant rewriting QuickDraw, the graphics library Bill Atkinson had written for the original Mac. The rewrite was slow, and a Radius programmer noticed.

One of the programmers that we hired, one of the very first ones, a guy named Ted Cohen, had just gotten a Mac II [...] and he noticed that the eight bit graphics in the Macintosh II were really, really slow. Just, you know, easy to see, maybe half as fast as they could, or should have been. (part 2, p.2)

Hertzfeld had no source code. He had a technique.

I had my own technique back in those days of finding the inner loops of things. Just hit the interrupt key, have it do the thing you want to measure, have it hit the interrupt key, and it'll, statistically, it'll be stopped in the place that it's using the most. (part 2, p.2)

What he found was a choice that made the code easier to write and slower to run. The Mac II used the Motorola 68020, which had new instructions that let a program address individual bits in memory. Convenient for the programmer; for graphics, a disaster, because every pixel became a separate trip to memory.

The 020 had these new instructions that weren't in the 68000 called bitfield instructions, where you can make it bit addressable. And while that's really kind of cool for ease of programming, it's horrible for the speed of the graphics. [...] Like, you have to fetch it in memory, muddle with it in the registers, and store it back, and for each pixel. So, really, what you want to do is hit memory with the full width of the data bus, so you have minimum, you know, memory fetches, and so, just fixing that made it three times faster. (part 2, p.2)

Then the harder problem: QuickDraw lived in ROM, and you cannot patch ROM. His first attempt copied the whole thing into RAM, which worked and was useless as a product. Disassembling further, he found that Apple had left a door open.

As I was disassembling, you know, figuring out what's going on, I saw they had a jump table for all the inner loops. Every inner loop was, at first, it was designed to be patchable. I didn't know this. I wasn't working at Apple at the time, but when I saw that it was designed to be patchable, would support an extension in a clean way, I thought, "Boy, this could be great for the Mac II, you know, to make it three times faster is tremendous." (part 2, p.3)

He calls this the single technical project he is most proud of. (part 1, p.42)

Now the part that has nothing to do with code. Smith wanted to keep the patch as a Radius advantage. Hertzfeld wanted Apple to own it, because a speedup this obvious would be reverse-engineered by every clever third party and the Mac would end up with a dozen incompatible versions. But he had just learned that a large cheque from Apple was a way to get software shelved, and the executive running Apple's product organisation was Jean-Louis Gassée, who in Hertzfeld's account "hated the original Mac team, and was trying to trip me up when I was on leave of absence, every way he could." (part 1, p.43)

The way through was a question.

I was a little bit suspicious of the management of Apple. In particular, Jean-Louis Gassée, didn't really like the original Mac team [...] I didn't know what to do, until finally, Jerome Coonen, who was the software manager, I asked him, "Well, what's the maximum you're allowed to sign for, so no one else would have to sign off on it?" He said, $10,000. I said, oh, fine. (part 2, p.3)

Ten thousand dollars for a patch that tripled the speed of Apple's flagship. It went to Bruce Leak, the engineer then responsible for colour QuickDraw, and shipped in System 6.

We started with less than a month before it had to go out, and we got the speed of the graphics tripled. So, I was proud of that one. (part 2, p.3)

Hertzfeld's own summary of the two sales:

It was because I had learned that lesson, that if they pay you a lot of money for it, they don't want to use it. (part 1, p.43)

The forecast nobody would cut, and the product she quit over

Management · Apple, 1985

Joanna Hoffman was the first marketing person on the Macintosh team, from before the product had a name. She rolled it out across the United States, then ran its international launch, and in 1985 moved back to domestic product marketing. What she walked into was a division still reporting the numbers it had promised before launch, a year after the market had said no.

The Mac had sold. It had not sold what Apple said it would. Hoffman is clear that the product's limits were the cause: too little memory, a small screen, and a printer situation she still finds hard to believe.

This is a product that sold essentially without a printer. The only thing it had was a dot matrix printer, because we wanted WYSIWYG and you couldn't do that with that letter quality printer, which was essentially a modified typewriter. [...] So, despite all those limitations it still sold quite a bit. It's just that our forecast had been unrealistic. So, it's the expectation versus the reality that was haunting us at the time. (Hoffman, CHM oral history part 2, p.13)

She was now the person responsible for reporting that forecast.

The first forecasting meeting I walked into, I realized that the domestic forecast hadn't changed since we had shipped. There still were these crazy unrealistic forecasts, so I had to just slash them, which was a major relief to our sales and distribution. But, of course, it was not well received by anybody else. But you can't fight reality. It was in our faces that we weren't selling as many as we had predicted, and yet we were still pretending like this was going to be happening any minute now. The forecast next month will be this gargantuan forecast. (p.13-14)

Asked how big the gap was: "It was more than 50 percent." (p.14)

Sales and distribution, the people who had to hit the number, were relieved. Donna Dubinsky, who ran distribution and had been challenging the division's forecasts for months, was on that side. The people above Hoffman, who had signed off on the original number, were not.

That was survivable. The Macintosh XL was not.

The Lisa was Apple's $10,000 business computer, launched in 1983, and by 1985 it was not selling. Mac users, meanwhile, wanted a bigger screen and more memory than the Mac could give them. Somebody, before Hoffman took over the product, decided to solve both problems at once: run Macintosh software on the Lisa hardware and sell it as the Macintosh XL.

It was a bit of a farce in that when you emulated the Macintosh, it wasn't a full emulation; only a certain percentage of applications could run. [...] It was slower because it was built on top of the Lisa. (p.15)

She saw the problem on arrival, and it was not the emulation. It was the plan.

The pent-up demand in the market was such that when I came onto the project, it had just been announced, and I told them that this is going to sell out in three months. And then we are in the deep doo-doo, because having a three-month product that we had no intention of supporting into the future was going to cause a major dissatisfaction in the marketplace. The forecasts that were presented based on Lisa's previous sales was that it would take eighteen months and it will slowly dwindle, and it would just peter out into the sunset with nobody noticing. (p.15)

Well, guess what? We sold out in less than three months! We had no intention of building more. And it was a major scandal except that it was overshadowed by a different scandal: Steve being kicked out of Apple. So, nobody noticed the XL fiasco. But I felt like I was put in a position to lie to the customers and I just didn't like that. So, I went on a leave of absence and then eventually I decided not to go back. (p.15)

Businesses had bought it. With the dot matrix printer. "Can you imagine?" (p.15)

Every project had a nine-month schedule and took three or four years

Management · Apple and NeXT, 1978 to 1990

Rich Page designed hardware at Apple from 1978, was made an Apple Fellow when the Lisa shipped, and left with Steve Jobs in 1985 to co-found NeXT, where he ran the hardware. He had watched Jobs set schedules at two companies over twelve years, and he had a theory about why they were always wrong by the same amount.

The story starts with a new hire.

I hired an engineer, he was very good, and I brought him on board in the spring of 1990, and he was going to work on the color NeXTstation. And I remember we're in the lab one day, and he had just started a few weeks earlier, and we're in the lab and Steve's there, and Steve's making a point: we got to get this done by October. So we have a discussion for a while and eventually Steve leaves. (Page, CHM oral history part 2, 59:25)

The engineer then tried to work out what he had just heard.

He says to me, "I don't understand. Obviously it doesn't mean this October, because that's impossible. But October next year is like seventeen months away, and you can't mean next year. But he can't mean this fall. So what's going on?" And I said, "Well, unfortunately, it means this October." (1:00:18)

A person three weeks into the job could see that the date was impossible. The people who had been there for years had stopped noticing. Then Page's larger observation, which is about Apple.

I always wondered, when I was at Apple, you realize that almost every project at Apple took three or four years to get out the door. But every project at Apple was started with a nine-month schedule. So why was every project a nine-month project, but they took three to four years? That's not off by 10, 20, 30 percent. That's a big factor. I never really came up with a good answer to that, but I had two thoughts. (1:00:52)

The first thought is about engineering.

I don't think Steve trusted engineering. I think he thought, whatever schedule I give them, they're gonna double or triple it, so I might as well give them a shorter schedule. I think that was part of it. (1:01:31)

The second is about money.

If you went to Mike Scott and the board and said, "I need nine months," they'd probably say yeah. But if you were a little more honest and say, "I need two and a half years," they might say, "Oh, that's too much money," and maybe they would have not agreed. So I think maybe he sold it to Apple based on the fact that, well, it's only nine months, we can afford it. And then when it took three years or four years, oh well. So I think it was partly selling it to other parts of management, and I think it was also defending himself from engineering. (1:01:48)

Mike Scott was Apple's first CEO, the person Jobs had to get funding from.

Page adds one more mechanism, about what happens after a product is frozen.

You get your product done, and at some point it's done on this day, but it doesn't go out the door for another three, four or five months. [...] The day you freeze the product, from that point on people start accumulating things they want to change about it. So then you ship the product, and it's six, nine months later, and somebody in engineering has a bright idea of making some small change. So what they want to do is open up the front, make some small change, close the product back up and ship it. That turns out to be nearly impossible, because the minute that other people realize that you're going to change the product, everybody comes out of the woodwork. (1:02:54)

Getting Steve Jobs to sign off on a rectangle of greys so the factory could stop asking him

Design · The NeXT factory, 1988 to 1990

The NeXT Computer was a black cube, and the cube was cast magnesium, painted. Rich Page ran hardware at NeXT and owned the problem of making that cube come out of the factory looking the way Jobs wanted, over and over.

A casting comes out of the mould slightly different every time, and then it has to be sanded, textured and painted, each step adding its own variance.

The NeXT cube was a magnesium structure, but painted. As it turns out, I think that was a mistake, because it's hard to take something that's cast [...] and have that whole process be repeatable. (Page, CHM oral history part 2, 33:07)

So every cube was a little different, and the person judging them had the final say on everything.

They got to get sanded and they got to get painted, and they all vary a little bit. You end up with Steve coming to the factory and saying, "Well, I don't like that one," because they were variable. (34:16)

Which meant the line could not run without him, and could not predict him.

Sometimes there was something he didn't like, and yet a week or two earlier it was okay. So you don't want to deal with the whims of "he doesn't like it." (35:25)

The fix was to make his taste into a document he could sign.

What we did at one point is we built a matrix of different textures and different colors, and maybe not dramatically different, but shades. And we got Steve to buy off on: okay, if it's in this rectangle, these shades of grey, or okay in this variation in textures, okay. Something outside that rectangle's bad, but if it fits inside here then it's gonna be okay. And it took him a while to buy into that, but that made life a lot easier, because then you didn't have to ask him. You just knew. (34:39)

For the next machines they changed the material so the problem went away at the source.

What we did later with the black-and-white NeXTstation and the color NeXTstation, we had these things called the pizza box. What we did is we did a cast aluminum structure with a plastic outer structure. And the nice thing about that is you get this plastic cover the right texture and the right color, and then it's easy to make lots of them. (33:44)

The research lab that worked while people were forced out of it, and died when they stopped being

Management · Apple's Advanced Technology Group, late 1980s to 1997

Larry Tesler came to Apple from Xerox PARC in 1980, worked on the Lisa, and later founded Apple's Advanced Technology Group (ATG), the company's research lab. He knew the standard failure of such labs from the inside: PARC had invented much of the personal computer and shipped almost none of it. Good ideas were never the problem. Getting them across the wall into products was.

ATG's answer to that was a rule about people, not projects.

Every year, 15 percent of the people in ATG would leave ATG and go to Product Development and take their ideas with them; join forces with other engineers who were sometimes more development-oriented than they [...] And they'd go in and they'd turn their ideas into products. And then we would take that same number of people out of the Product Engineering Group, which was about 10 times bigger than ATG. So that would be maybe two percent of their engineers and bring them into ATG. And that way ATG stays the same size. (Tesler, CHM oral history, p.39)

Who came in mattered as much as who went out.

It tended to be engineers who were burning out. They had five projects in a row that had no space between them and they were long hours and they just wanted a breather. They wanted to try out some ideas they've been thinking of for years and never had time for. And so it was a very strong connection. It was based on Gordon Bell's observation that the only way to transfer technology is to transfer people. (p.39)

The lab had a rule about scope as well: a project had to have a credible path to product in eighteen months to five years, or it did not get started. And it had enemies.

Some of the Product Development managers were idea people and they didn't like the fact that before they even got a chance to come up with their way of doing multimedia we were coming up with a way to do multimedia and sending it to them half built and ready to productize. We thought it was great. [...] But the engineering managers and some of the engineers in Product Development felt like it wasn't fair. They were doing the hard work. We were doing, you know, fun work and we were getting to define the next generation of stuff and it wasn't right. (p.39-40)

In 1990 Tesler moved to the Newton project and handed ATG to David Nagel, one of his direct reports. Engineering management had already attached a condition.

The Engineering Management team basically had made a proviso in there somewhere that we were going to review this policy of this 15 percent and this kind of what they felt was a monopoly of new ideas having to come from ATG. And sure enough, right away that changed. [...] The main thing I remember is that they decided that ATG should have a longer horizon. [...] They wanted to move that 18 months further out and make it be more like three years, five years, you know, and like five to ten years. (p.40)

Then the budget cuts of the early nineties.

Even though they were transferring people out of ATG they weren't having the budget to hire new people and transfer people into ATG. The result of that was that ATG became more and more research-oriented. It was the people who didn't ever want to transfer to product development who were still in research and it was kind of like a university kind of department. We're doing things for the long term. They should turn into papers. They get published. They should inspire people rather than become product templates and Apple could ill afford that in the 90s because [...] the money was no longer rolling in and they were running out of ideas and this group wasn't giving them useful ideas anymore. All the useful ideas had already been transferred and they couldn't come up with new ones because they couldn't hire new people. (p.40)

Early in 1997, with Jobs back as an adviser and Amelio still CEO, Avie Tevanian, then running software, asked Tesler to go back.

"We're shutting down ATG. People say, oh no, we're going to lose a lot of great ideas. Since you started it, would you go over there and find out what the great ideas are that are still there and which ones we should save and which ones we should kill?" So I was sent back there to kill the organization I had founded, but I'm glad he did it. It was the right thing to do. When I got there, I couldn't believe what I was seeing. They were all really cool, but they were irrelevant. (p.40)

His example is a touch-screen conference table with displays under the glass, which he dates as fifteen years ahead of Microsoft's Surface table.

We could sell 10 of these at a time. This is a company that comes out with products to sell in the hundreds of thousands and the millions, not in the tens, wrong company and so I killed that and I killed a few other things, spun out a couple of things. (p.41)

He kept three: a handwriting recogniser for the Newton, which was still alive; QuickTime for Java, cancelled for one day and then reinstated; and a hard-drive search project that became Sherlock, the ancestor of Spotlight. (p.41)

API review with no approver: a mailing list, a one-week clock, and three abuses in fifteen years

Engineering · NeXT and Apple, 1989 to 2011

Bertrand Serlet came to NeXT from Xerox PARC in 1989 to work on AppKit, the framework every NeXTSTEP application was built on, and ended up running all of Apple's software engineering through the Mac OS X years. The thing he brought with him from PARC was a review process for public interfaces.

Why an API needs review at all, in his words:

You can change the code, the implementation, but changing the API, now you need to evolve all the apps that have depended on it. It's very difficult. It takes years, literally. [...] You tell people, I'm going to start deprecating, then you deprecate it several releases. It's typically two, three releases. (Serlet, CHM oral history, p.35)

The usual response is a heavy gate: a committee, an approver, a sign-off. Serlet went the other way.

I was surprised that there was no API review when I came in and the NeXTSTEP API could have been better if they had had a review and that was one thing that we did with OpenStep, we started doing reviews [...] and it's very delicate because you want the owner of the API to own it. You want owners to own, right? You don't want to disempower owners, but you want to have feedback too. (p.34)

The process, in full:

When you have a new API, that's public API, that customers will, developers will use. You have to publish it to a certain forum and you have to address all the feedback, one way or another, you can say, oh yeah, I don't care about that. But you have to address the feedback somehow and it's time-bound. It's a week or some limit like that and after that, that's it. You can go ahead. Your API is approved. (p.34)

The forum was a distribution list.

We had the forum be a distribution list of all the folks that are API savvy and pretty much the trick there is that anyone who has to be on the list is on the list, and that self-determination really works, because people who are not interested in this will not ask to be on the list, right? (p.34)

On abuse:

It's subject to abuse. I think during my tenure at Apple over 10 years, right, 15 years, there were two or three cases of abuse. In those cases, that is the feedback was not addressed or the feedback was this is really a bad API and they went ahead, and you deal one-on-one with the abuse. Very minor. Two or three out of thousands of API reviews. So I'm a big proponent of very light-touch processes, not heavy processes and leaving people ownership. (p.34)

Running the Intel transition with no schedule, only a punch list that had to get shorter

Management · Apple, the switch from PowerPC to Intel, 2005 to 2006

Bertrand Serlet ran Mac OS X engineering when Apple moved every Mac from the PowerPC processor to Intel. A processor change is the largest thing that can happen to an operating system; every piece of software has to be rebuilt, and much of it has to be rewritten. Apple did it in six months, half the time it announced, and Serlet's account is of two management decisions that made that possible: one taken years before, and one about what to tell the team.

The first was Avie Tevanian's, and Serlet enforced it.

One thing that Avie pushed, and I enforced it, is that we always compiled our code for Intel as well as PowerPC, even though we had no Intel machine and so we had some checkers in place in the build system to make sure every single project can be built for Intel and this is years before we did the transition. (Serlet, CHM oral history, p.23-24)

The deal with Intel was signed in February 2005.

We just scrambled to make it happen for the developers conference that was in May, so three months later [...] We wanted also developers to have machines that they can play. But we didn't want the Mac OS to run on any PC. So there was a delicate balance. So we decided to build machines in secrecy and in fact, we enlisted Simon Patience's team, which was the CoreOS team, to actually build the machine that we were going to give for developers at the conference. So for a while, the kernel team was actually building, assembling machines in a secret lab in preparation for WWDC and it was the kernel team, because they were the first one to help make it work. So they all were disclosed. (p.24)

Then the schedule.

We said at that time that it would take about a year to transition and nobody believed it. All the industry thought that it would take much longer. We were hoping it would take less. But we were not sure, because we had a dependency on Intel for some of the new chips coming up. So we were not sure, and still we're not sure, for several months. So I said, well, we're going to pretend we need to ship ASAP and we don't have a schedule, but we just have a punch list of what's left to do and we're going to shrink the punch list and that's what we did. People were upset because they wanted to know the schedule and I said, sorry, I can't tell you the schedule, I don't know the schedule, but it's ASAP, and so we were able to ship in January with the new Intel Macs and so that was six months, not a year. (p.24)

Then two worries.

Now that we were on PCs, people could compare the speed of the Mac on what was essentially an Apple machine, but a PC, so same processor, with Windows NT and so I was very worried by that [...] So I said, let's do benchmarking of lots of things that analysts could benchmark or end users could benchmark. For example, you open 100 Windows, how long does it take, right? That kind of thing and we wrote several hundred tests that we could run both on NT and on the Mac and the results came in, and for 95% of them, the Mac was slower. So we had a big communications meeting. I motivated the team by saying, hey, not so good, and by the end of summer, we had flipped that 95%. (p.24)

The second worry was about the year in which both kinds of Mac would be on sale.

People will find some subtle differences between them, because we've still evolved the Intel code base [...] And that people would play the game of find the difference. So, which I think would be very damaging for moving forward. I wanted the transition to be viewed as invisible. So what we did for the rest of that year is we did software updates that had all the improvements that we made to the Intel side. But that also were all those improvements that were not for PowerPC, but we also included that in the PowerPC update and so we had a single code base. We forced the code base to be the same through software update. So by the time January came in with the new Intel machines, they were exactly the same software as the PowerPC. So no one ever found some bug difference between the two. (p.24-25)

Deciding which APIs to kill by scanning Microsoft Word to see what it would break

Engineering · Apple, the Carbon transition, 1998 to 2000

Nitin Ganatra joined Apple in 1993 answering developer questions in technical support, and by the late nineties he was on the team building Carbon. Carbon was the bridge: Apple had bought NeXT and was replacing the classic Mac OS with a new system built on NeXT's foundations, and every existing Mac application was written against the old programming interfaces. Rewriting for the new ones would have meant shipping an operating system with no Photoshop and no Word on it. Carbon was the subset of the old interfaces that would keep working on the new OS.

Deciding what went into that subset is the story. The instinct was to cut.

We wanted to throw out as much as possible. [...] Because everything that you throw out is just, every API you throw out is an API you don't have to support down the road. Bertrand [Serlet] later, you know, sort of captured that sentiment in a great way by saying that APIs are a liability, you know. They're an asset to your company, but they can also be a liability. The more APIs you have, the more exposure you have to programs that may not work well or make assumptions based on the behavior of those APIs. (Ganatra, CHM oral history part 1, p.28)

The old interfaces were a liability for a specific reason: they had been designed for a machine with 128K of memory, and they showed it.

In the old days, data structures were sort of fully defined and fully fleshed out as part of the definition of an API call. And part of the reason for that was [...] you really don't want to create multiple calls to get and set things out of a data structure. Really, the more resource friendly way of doing that would be to just say, "Here's the data structure, you can get, and pull, you know, get and set things in there all you want." [...] We only have 128K of RAM to run in anyway. So let's just do away with that and just expose the underlying fields and just let the developers do what they want. So fast forward 15 years from, you know, from that attitude, and really, it does become more important to hide the underlying implementation so that you can change it later. (p.29)

Cut too much, though, and the developers would refuse to port.

We had a lot of good data by then where we had these tools where you could scan an application, you could scan a popular application and look and see what are the APIs that it's actually using. And so from that, you know, you could bring up Microsoft Word and run it through these tools and it would say, "Microsoft Word is using 2000 API calls. Here they all are." You know, and you could even flag [...] APIs that we really want to get rid of these ones, but we don't know if we can yet. So let's run through the list of popular apps and see, by getting rid of that API in practice, who are we hurting here? (p.28)

The target Ganatra gives for how far to go:

It was sort of this riding this line between giving developers everything they want and, you know, tying our hands for [...] future development work, or, you know, dial it back as much as we can until developers almost scream that it's too much work to do, but now we're set up well to support them in the future, and just sort of trying to figure out where that was the whole time. (p.29)

Not telling the designers what the hardware could do, on purpose

Management · The iPhone, 2005 to 2007

By the time the iPhone project started, Nitin Ganatra was managing application teams on Mac OS X, and he became the director responsible for the phone's built-in apps. The designers he worked with were Apple's Human Interface team (HI), and the hardware they were designing for was an ARM processor with a fraction of a Mac's memory. The obvious way to run that relationship is to tell the designers the limits up front so they do not waste time on things that cannot be built. Ganatra describes doing the opposite, and why.

Really one of the goals was to sort of let the HI team do, come up with, the best possible design and not bog down the HI team with discussion about number of colors on the display or how much RAM we're going to have or how much, you know, this, that [...] obviously as an engineer you're thinking about those constraints all the time and you're thinking, "Holy crap, am I going to be able to implement [...] this really cool thing that the HI team came up with?" But really you want to, you don't want to start to burden the HI team with these constraints early on. (Ganatra, CHM oral history part 2, p.10)

His reason:

Because then the best thing that they're going to ever come up with is a design that they themselves, the HI team themselves, anticipate will work given all the constraints that were given. You've now put the HI team in a situation where they're now trying to guess at whether their design is still really cool enough to be compelling to a customer but still something that can be implemented. (p.10-11)

It's better to just sort of stand aside and let HI come up with the best possible designs [...] and then we'll figure out if we need to, you know, if we need to implement something that's 90 percent as cool but is possible, on an ARM-embedded device or what have you. We can have that discussion later, but you come up with the best design, the engineering team will come up with the best implementation that we can of that design, and if there isn't [...] a complete match between the design and what can be implemented, by then we'll have a better understanding of what the details are, where it's difficult to implement things, and we'll have, on the engineering side, we'll have specific recommendations for, "Well, yes. You said that this thing could be an unlimited length table, but what if we made it a thousand cells instead?" (p.11)

The same approach shaped the software underneath. Rather than design a grand framework for moving between screens before anyone knew what the screens were, the team built the first apps directly and let shared code accumulate.

If you try to create this architecture too early on, you can kind of stifle the ability to go in different directions or the ability for [...] the HI team to kind of go and go hog wild [...] And so, it's just better to kind of just let it be organic for a while but understand that there's a plan there. (p.10)

That accumulated code became UIKit, which every iPhone app has been built on since.

Three days of memorising a design in one room and redrawing it badly in the next

Management · The iPhone, 2005

The iPhone was built under a secrecy regime in which Steve Jobs personally approved two lists: who was on the project, and who was allowed to see the user interface. The two lists were not the same. Nitin Ganatra, who managed the team writing the phone's applications, was on both. The engineers who reported to him were, for the first few days, only on the first.

For a few days, I had disclosure to see the UI as the manager, but the engineers on my team had not been disclosed yet [...] It was probably for the first three days or four days that we were in this arrangement where Steve Jobs had to personally approve not only everybody who was on the team but then also everybody who had access to the user interface. And so, and I remember I had discussions with Scott Forstall about this too, where I had access to see the designs but my engineers who are actually doing the work here didn't have access. So what should I do? You know, like, should they just go on vacation for three days? That doesn't seem right. (Ganatra, CHM oral history part 1, p.64)

Scott Forstall ran iPhone software. There was a demo due that Friday or the following Monday. The solution was a workaround for a rule neither of them could change.

We had two of my engineers were in one office and in an office right next to them was a computer that had the designs on it. And so, any time they had a question about the designs or some aspect of the design itself or something, what I would do is go into the, and they weren't allowed in the room with the computer that had the designs on it. So I would go into the room with the computer that had the designs on it, play with it for a little bit, try to commit to memory everything that I saw, and then I would come back over and on the whiteboard I would draw, and I'm a terrible whiteboard drawer anyway. And then I would draw an approximation of how things worked and how it behaved and how things should be. And so, we did this for three days. And the whole time we were doing it, we were all perfectly aware of how absurd it was. (p.64)

The interviewer's comparison was to a corporate spy inside his own company. Ganatra's summary was shorter: "It was a little silly for a while there." (p.64)

Antennagate: asking for the complaint data before agreeing to apologise

Marketing · The iPhone 4, 2010

Regis McKenna ran the agency that positioned the Apple II, shaped Apple's marketing from the beginning, and knew Steve Jobs for thirty-five years. By 2010 he had twice declined to come back to Apple, preferring, in his words, to stay friends. So when the iPhone 4 shipped with an antenna that lost signal if you held the phone a certain way, and the story had a name, he was called in as a friend rather than a consultant.

I got a call from Steve, and he was actually in Hawaii with his family. He said he was going to fly back and could I meet him on, I think it was Monday or something like that, and I said sure. He basically called together a meeting of some of the people from Chiat who were still working on his account. He had his marketing people there, and some of the engineering people. And we had a meeting in his conference room, and his son, Reed, was there. He wanted him to observe it. (McKenna, CHM oral history part 5, p.12)

Chiat is Chiat/Day, Apple's advertising agency.

I asked for the data. I wanted to see what the data had to say, because I knew Apple, they had collected a huge amount of data from all the users and all the people that had complained, all those who wanted returns on the phone, and so forth. The iPhone before [the 4] was actually worse than that phone. So it was really a relatively minor issue to the consumers that were buying it, from my standpoint. (p.12)

The previous model had measured worse on the same problem, and nobody had noticed.

I first advised him to just let it go. I said, "It'll fade away, fix it." He didn't want to do that. He said, "No, we've got to address it." I said, "But there are only, really, a relatively few people that are making noise about it. You may be making a bigger issue." We talked about that in the meeting. (p.12)

Jobs overruled him on whether to respond. On how:

Most of the other people were telling him to come out and apologize. They really wanted him to put his tail between his legs and go out there and be very humble and so forth. I said, "Don't do that. I don't think that's you. I don't think that's what you should do. And I don't think the data calls for that, quite frankly. I think you do tell people that technology products are never born perfect, that they will improve, and they're constantly being improved." And he used that line, actually, when he talked. Basically, he said, "We will address the problem." They gave you a little protective band that you put around the iPhone. And, he said, on the next generation, as soon as we can, we'll change it. (p.12)

Once he went out and said these things, they [consumers] didn't lose confidence in the company by saying, "Oh, here's a big failure." The president of Microsoft at the time said this was going to be a failure, this product. And it wasn't. It still continued to sell. The next generation sold even more. (p.12)

Turning the strongest objection into the objector's assignment

Engineering · Apple, the Mac OS X transition, 1997 to 1998

When Apple bought NeXT, Avie Tevanian, who had written the Mach kernel at Carnegie Mellon and then run software at NeXT, became the person deciding which technology survived: Apple's or NeXT's. Every one of those decisions was read inside Apple as a side winning, and this is his account of the one that produced the most noise.

Apple had its own networking system, AppleTalk, which had been designed a decade earlier for small office networks and did one thing better than anything else: you plugged a printer in and it appeared. The rest of the world had standardised on TCP/IP, the protocol the internet runs on, and Apple's own TCP/IP support was bought in from an outside vendor.

The networking standard for Apple at the time was AppleTalk, and AppleTalk was just wonderful networking for a LAN [...] It was way better than TCP/IP. But, the rest of the world has standardized on TCP/IP and it was obvious to me and some others who weren't totally immersed with the Apple way for many years, that TCP/IP was going to win all the battles. It was already winning the battles, and at the time Apple had an AppleTalk team working on AppleTalk, and their IP stack was outsourced, and so I said, "This doesn't make any sense. Networking is a key part of the product." (Tevanian, CHM oral history part 2, p.15)

His proposal was the open-source stack everyone else used, which happened to be the one NeXT already shipped. That was enough to make it an Apple-versus-NeXT fight.

I got no end of complaints about that. [...] In fact, people went around me all the way to Steve saying, "We're going to tank the company if we do this," and including the people who are selling us the stack, the software stack from the outside. They were making money off Apple, and so I just sat down and I laid it out and said, "This is the standard of the future, number one, and number two, this is the standard implementation, number three, it's free and open source. People will keep working on it. This is [a] no-brainer decision. End of discussion." (p.15-16)

Winning with Jobs settled the decision. It did not settle the engineers, and the objection they brought him was a good one.

Good engineers would come to me and they would say, "but you can't do this. Our customers know and love AppleTalk. They just plug in a new computer and it just works and that doesn't work with TCP/IP. It works with AppleTalk," and they're, plug in a printer. It just works. I said, "Great. So your job is to now make TCP/IP do this." (p.16)

Lo and behold, here we are today, 20 years later, using TCP/IP. Everybody just plugs in their computer or their printers and devices and everything and it just works and that's because we got motivated to fix a few problems and fill in a few gaps with TCP/IP that didn't allow it to do that before. (p.16)

He also gave them the arithmetic, and says how it came out.

"What you're advocating is a solution that is going to keep three percent of the addressable market happy." That was the existing customer base. Maybe it's five percent. [...] "I want to keep those people happy but I want to have a way to go get the other 95 percent, and we can't get the other 95 percent with something like AppleTalk. They're just never going to buy it," okay, and, you know, we never got the other 95 percent, but you look today and Apple's doing pretty well even with computers. (p.16)

Not everyone made the switch.

There were some people, they just couldn't get their head around this way of thinking and they either quit or were fired, and then the people that could, they said, "You know what? There's a few things wrong with TCP/IP today, but we know how to fix that, and we can work with the IETF and change the standards and evolve it and do a few things with DHCP and other standards and fill in here and fill in there and make some DNS changes and we can make this all work." And, it does. (p.16)

"Over my dead body," and the two products that argument shaped

Product · AirPort and the iPod, 1999 to 2004

Jon Rubinstein ran Apple's hardware engineering from 1997 and later the iPod division. Two of his stories are one story: an argument he lost about Wi-Fi, and how losing it changed the way he fought the same argument about the iPod three years later.

AirPort was Apple's 1999 Wi-Fi product, a base station and a card, at a time when Wi-Fi was one of two competing standards. Rubinstein had spent time in Washington keeping the other one, Intel's HomeRF, from getting the rules changed in its favour, and when Wi-Fi won he saw what Apple had.

Wi-Fi became sort of the de facto standard and I'm looking at this and I'm going, "This is a big business." Right? Our base stations could be used because we're just a PCI card, right, so we could take our PCI card that goes in our Mac, plug it into a PC. We could take our base station. We're missing one thing, and that's the application that runs on a PC to configure the base station. It's not a big deal. (Rubinstein, CHM oral history part 1, p.81)

He took that to Jobs.

So I go to Steve, and I said, "Look, I need three, four people. I'm going to port our custom app on the Mac over to a PC version, so we can enter the base station business on the PC." Steve goes, "Over my dead body." I said, "But Steve, it's like three, four people. I mean, we already have it running. We just need to productize." "No." All right. So I'm like, "All right." "No." I mean, I had lots of other things to do, so... (p.81)

Jobs's reason was that AirPort was part of what made a Mac worth buying. Rubinstein's counter was that Apple's base stations were better than anything on the PC side, with features nobody else had.

We had mesh. No one else had mesh. Until Eero came out recently, no one else really did mesh. We had that. We had easy configuration. There was no SSID, right? I mean, it was the name of a network. I mean, it was Mac-like, right? Everything was simple. [...] I said, "This is going to be a multibillion, I want to build a multibillion-dollar networking business." Right? And Steve went, "No way. We're not doing that." I said, "All right, so we're not doing that." So we kind of put that aside, right, and, which then came when we did the iPod later on. I dug my heels in. (p.81)

The iPod shipped in 2001, Mac-only. To get the music labels to license songs for the iTunes Store, Jobs told them the Mac's market share made it a safe experiment.

Steve convinced the music companies that, "Look, this is a great experiment because at most we've got 2% market share. At most we've got 2% market share, so you do not have to worry about 98% of your market, and so it's a great way to experiment," and so they went "Okay." And frankly at the time we were going to stay Mac only. We didn't have this idea we were going to expand to the PC. (p.92)

Then the same argument as AirPort, with a different outcome.

It makes people buy more Macs, and that was the whole point of the iPod, was to sell more Macs, but the lesson from AirPort just sort of sat on my shoulders and sat on Phil Schiller's shoulders, and we started hammering Steve. "We got to take this stuff to the PC. We got to take it to the PC," and we just kept haranguing him, and finally he goes, "Fuck you guys. I don't want to talk about this anymore. You do whatever you want." (p.93)

What they did with it:

So we went out, and we found a software company. I think they were in L.A. [...] We basically hired them to take their PC software and hook it up to the iPod. Music Match is what it was called. [...] They'd done a product that was sort of a half-assed version of iTunes, and it wasn't very good, but it worked on a PC, and it worked with portable players, and so we just modified it to work with the iPod, and then we gave them a very strict roadmap, and we said, "Look, here's the roadmap we want, and this thing's going to put you guys on the map, so here's what we need." And Phil and I managed them very closely and managed their roadmap and the iPod on the PC. (p.93)

The same passage carries a second reversal.

By the way, we also did the Mini at this point in time, which Steve tried to cancel. I got a panicked phone call one day. "You better get to this meeting. Steve just cancelled the Mini." And I'm like, "Ahhhhhh," so I come running down there, and Steve's just leaving the room, and everyone goes, "What do we do?" I said, "Just keep going." Because the Mini is really the product that caused the iPod to take off because of the price point, and it was the anodized aluminum, the colors, [the] price point. It had enough storage but not too much. (p.93)

The iPhone developer story that lasted from a Friday to a Monday

Product · The iPhone SDK, 2007 to 2008

In June 2007, weeks before the first iPhone shipped, Steve Jobs told developers at Apple's annual conference that they could write applications for it using web technology, running in Safari. No native software; the phone's own apps would stay Apple's. By October Apple had announced a native SDK instead. Four people who were inside that reversal describe it, and they do not fully agree.

Richard Williamson ran the iPhone's Safari and WebKit engineering, so he was the person best placed to argue for the web approach, and had already stopped believing in it.

I and the iPhone Safari team had a pretty deep background in WebKit and web technologies. And so, I was in a position to try and advocate for that, although I had already come around to the idea that [...] native development was far more efficient than web technologies because of the tools, because of the fragility of web technologies, and [...] if you look at the Objective-C frameworks, they kind of guide you towards good development, guide you towards how to build an app. And HTML was never designed to do that. (Williamson, CHM oral history part 2, p.27)

On where the web-apps announcement came from:

Within the engineering organization, there wasn't a final approach to say, this is what we're going to do for developers. In fact, it was only a few days before the announcement that Steve made [...] that we heard that there was a great developer story. And really Scott [Forstall] and Steve huddled and came up with this. It wasn't vetted by me, certainly, and I don't think Nitin or Henri. So, we had to really scramble to try and figure out what that meant. And I think it was pretty clear to the engineering team that it wasn't going to be a great solution. (p.27)

Scott Forstall ran iPhone software; Nitin Ganatra and Henri Lamiraux ran its applications. Williamson's read on why it happened:

I think Steve said to Scott, "Look, we've got to do something about third party developers. What are we going to do?" And it was one of these snap things that Steve just wanted something and had to have something. And Scott came up with the story. (p.27-28)

Ken Kocienda, who built the iPhone's keyboard, remembers how long the story lasted inside the building.

I do recall there was like a Friday afternoon where some Safari and WebKit people from the outside team came to our iPhone hallway and were looking around at offices. "Oh, I'm going to sit here. I'm going to sit here," because they were actually going to come and do this web development story. And that was like Friday. And then Monday, it was like redone. It was like a very quick turnaround that we're going to do this story. We're going to do this story. We're going to do this story. We're not going to do this story. (Kocienda, p.28)

Ganatra was not in the room for the decision, but he describes the argument that did the work, and it was not abstract. They took one real application and tried to build it the way they were telling developers to.

Our very first, you know, our first target was Epocrates. And we were kind of thinking, "Well, what if there was a doctor and they're walking around with a Palm [...] Treo [...] and they were running Epocrates? Well, what would they do?" And so, there was some internal work around just kind of going through the exercise. Well, what would it mean for Epocrates to actually ship a web app? And I think it became, very quickly we realized [...] they would have to do an enormous amount of work [...] and by the way, once they had done all of that [...] what they would end up with, was an app that behaved nothing like any other app on the system. It would probably take longer to launch. It probably wouldn't have the nice smooth scrolling. (Ganatra, CHM oral history part 2, p.43-44)

Epocrates was a drug reference that doctors ran on Palm handhelds, exactly the kind of customer the iPhone needed to take from Palm. Ganatra's summary of the exercise:

"Well, what if we did want to go and target this client, and wanted them to make a really great web app, what would they have to do?" Answer: "An enormous amount of work, and they'd end up with something not great anyway." So, I mean, you know, I think we went through enough examples like that until we realized that really the right answer here is to release an SDK. (p.44)

The general principle he draws is about having two technologies.

If you've created two technologies, one for your own use, and one for third parties, or for somebody else to use, then that must have been for a reason. It must be that either the things that you're doing are so different from what you expect third parties to do that you need to have some brand new [...] technology that's different, or you think that what a third party will be doing or could do is not as important or as compelling or interesting as what you may be doing on your own. So that's why you have two. In either case, it's not a great answer. (p.43)

What made the reversal affordable was something Forstall knew about his own organisation.

Scott Forstall, to his credit, he knew what we were, you know, what these projects looked like and how we were already building the software. He had a good understanding that, internally, even though we didn't have a third-party SDK, internally, we were developing these things as though we had, you know, using our own internal SDK. So then the amount of work that we would have to do, we already have an SDK, basically. So, we would have to do some sanitizing and cleaning up and getting some interfaces ready to share with the outside world. [...] You're going to be happy with those interfaces for ten or fifteen years or twenty years. (p.44)

Bertrand Serlet, who ran Mac OS X and argued for the web side, adds the precedent.

A few years prior to that, we had on the Mac, a feature called dashboard widgets. That was very, very small, lightweight apps [...] We came out with dashboard widgets and within a couple of months, we get tens of thousands of dashboard widgets and so we knew that there was pent-up demand for lightweight applications on the platform. [...] So we argued how to open up, web app versus Cocoa. We did Cocoa, and we opened up and a year after, we started getting apps and I thought we'd get thousands of apps based on this dashboard widget experience, within a few months. I was totally wrong. We got 100,000 apps. (Serlet, CHM oral history, p.29)

Serlet also notes that Jobs "kind of liked the fact that all the apps were Apple apps" and "was actually arguing for a while that maybe we should keep it closed." (p.29)

The owners of the framework argued against using their own framework

Engineering · The iPhone, 2005

When the iPhone project started, the obvious way to build its software was to take AppKit, the framework every Mac application's interface is built on, and put it on the phone. AppKit was mature, the engineers knew it, and it embodied the Cocoa programming model that had come from NeXT and that Apple had spent a decade getting developers onto. Nitin Ganatra, who was about to manage the phone's application teams, describes the meeting where that option died, and who killed it.

The first option was to actually bring AppKit itself, which was the key component of Cocoa and make that the phone. (Hsu, interviewer, CHM oral history part 1, p.57)

The people making the case against were Ali Ozer, who managed the AppKit team, and Kristin Forster, an engineer on it. Ganatra's account is as much about engineers in general as about them.

Engineers are very sort of proud of the work they've done and they want to, we want to, believe that the things that we've created are, can be suited or can be used for anything, including things that we never anticipated they be used for. And a lot of times that's just not true, you know. But I think, some amount of the pride of ownership and pride of development kind of clouds your judgment in some ways and makes it so that you, you might think that the thing that you've developed is far more capable than it actually is. But that's the, but to Ali's credit, that's not Ali. And that's not Kristin. (Ganatra, p.58)

What they knew that nobody else in the room could have:

They were well aware of all the ways that the event system and that menus and that mice and that, you know, all these things that were very essential to how desktop computers work, they were so embedded into how AppKit handled events and did so much of what it does that they just, they themselves, Kristin and Ali, were making the argument that it's just not well suited for something like touch or for multiple layers on the screen or things like that. (p.58)

And why that settled it:

The fact that they were arguing against using AppKit to me was the strongest argument that okay, I mean, these are, they're super smart people and they are, and they understand their technology better than anyone. And so why would we try to use AppKit when the experts are saying we really shouldn't. (p.58)

Ganatra adds that Forster "did the majority of the research as well." (p.58)

The alternative was to keep the Cocoa programming model and write a new framework for touch, which became UIKit. Every iPhone application since has been built on it. The decision to build it rather than adapt what existed was made by the two people with the most to lose from that answer, on the strength of research one of them had done into her own code.

Choosing a browser engine: 150,000 lines you can hold in your head over 1.5 million you cannot

Engineering · Safari and WebKit, 2001 to 2002

In 2001 the Mac's web browser was Internet Explorer, which belonged to Microsoft, and Apple wanted one it controlled. Don Melton, who had managed part of the Mozilla project at Netscape, was hired to build it, with Ken Kocienda as his first engineer. Neither had built a web engine. Then Richard Williamson arrived, back from a year travelling, having asked Bertrand Serlet if there was anything interesting to work on and been offered two things.

He told me about two things. The browser project, and he said, "It's just starting. We're trying to figure out what to do." And then the other was, there was an effort that was super-secret at the time to go from PowerPC to Intel. [...] Of those two things the browser sounded far more interesting than working on the Intel project. So I said sure. And I didn't know anything about how to actually build a Web engine or a browser. (Williamson, CHM oral history part 1, p.24)

The question was never whether to write an engine from scratch; that fell away fast. It was which existing one to start from. Kocienda describes the state of the decision when Williamson joined.

Don and I were floundering around for a little bit of a while for a few weeks before Richard came on board. And Richard very, very quickly said, "Now, what are you guys doing?" [...] We were looking at Mozilla. We were looking at maybe licensing from Microsoft, Internet Explorer, for Opera versus iCab versus, there's KDE and GNOME, these Linux desktops or whatever. And Don and I were just kind of tripping over ourselves. We didn't really know what to do. Too many options. And so Richard goes away and just says, "How about KHTML?" (Kocienda, p.25)

KHTML was the engine inside Konqueror, the browser of the KDE desktop for Linux. Almost nobody outside that world had heard of it. Williamson went away for two days.

He calls us in for a demo and says, "Hey, guys look." And he's got KHTML working on a Mac. [...] He just convinced this KHTML code that yeah, yeah, yeah, you're running on a Linux machine even though it's a Mac. No, its X Windows. It's fine. Don't worry about anything else. And we were floored. (Kocienda, p.25)

Then the arithmetic.

Mozilla was the leading candidate at that point, the other leading candidate but it was a million-and-a-half lines of code. And KHTML was 150,000 lines of code. [...] It was three guys and we figured 50,000 lines of code each. (Kocienda, p.25)

Williamson had looked at Mozilla's code.

Beast. I think an important thing in software development is that the software does what it needs to do well and doesn't do things that it doesn't need to do. And the Mozilla code base has so much stuff in there that's irrelevant for a Web engine. Interesting technology in other domains but really irrelevant in terms of building a Web engine. And in order to get your head around a piece of software you need to be able to context switch it into your head and really understand how it works. It's almost impossible with Mozilla because of all of these irrelevant subsystems. (Williamson, p.27)

Kocienda had tried the other path and has the numbers from it.

Mozilla source code base had 20,000 typedefs. So you want to know what is in that software, you've got to learn 20,000 names. [...] I actually did get Mozilla running on Mac OS X which was a weeklong, which was a horrible, horrible trial because Mac OS X was so new that Mozilla didn't have a port. And this was secret. So you couldn't ask anybody. So I had to figure it all out for myself. So I got Mozilla running on Mac OS X but it basically never, I don't think it ever rendered a webpage. It would just crash. [...] So after a crash I'd look at the core file. It's 100 levels deep, 100 frames, 100 stack frames deep. So how are you going to context switch this into your mind? (Kocienda, p.28)

Opera was a serious option too, more advanced than KHTML, and they had a trip scheduled to go and talk about licensing it. They never took the trip. And Melton, who had managed Mozilla, "knew Mozilla too well to want to work with it again." (Kocienda, p.26)

The decision still had to get past Avie Tevanian, who ran software. Melton set one condition for the pitch.

Don said, "Okay, we're going to do the presentation but we're not going to use PowerPoint or any other application. We're going to do the slides in this application. And then we'll project that." So that's what we did. We actually made HTML slides and presented it in the browser and it was flawless. And we didn't tell Avie until the end of the presentation [that] we had been using the application. (Williamson, p.28)

One more thing was decided at the start.

We were hired to make a browser app that we could go and replace Internet Explorer as the double clickable app in the Dock. But we also from the very, very beginning we needed to make a framework. We needed to make a developer toolkit. That was part of the plan from the very, very beginning. And so WebKit was not after thought. (Kocienda, p.27)

Asking the customer to write a letter so he could win an argument with his own investors

Management · Stepstone, mid-1980s

Objective-C did not start at NeXT. It was created at a company called Stepstone, and Steve Naroff joined Stepstone to work on its compiler in the mid-eighties, before NeXT was a customer of any size. The compiler he found was, in his description, a translator: it turned Objective-C into C without understanding much of it.

It was naive. It wasn't full-bodied. It was not capable of detecting programming errors that are common, even a misspelling. Part of the reason it wasn't detecting errors was not only the technology but the language definition. There was no explicit interface declaration whatsoever. So if you typed a message expression and sent an object a message of a certain name and typed it incorrectly it would just assume that that was the name of the method. (Naroff, CHM oral history part 1, p.18)

Misspell a method and the compiler would agree with you. Naroff wrote a proposal to fix the basics: interface declarations so errors could be caught, a compiler that processed the whole language, and generated code that worked with the standard Unix build tools instead of silently producing broken executables. He could not get it funded.

I fought pretty hard for fixing the basics, which, to me, it was as obvious as [...] yet I got pushback from the venture capitalists who were thinking in dollar signs, thinking, "Why should we pay you to recraft and add some of these things that will make it more robust?" To them, error-handling, they're money people. This notion of, "Well, we can't flag an error," to them, that's like, "Well, so what?" They're not programmers. So I was getting a little bit frustrated when I was basically being somewhat ignored. There was some appreciation but this person, Ken, who was leading Software, he didn't have the technical chops to, as the guy who was running Software, to fight for me. I had to fight myself. (p.19)

What changed it was a visit from the customer.

The great fortune was NeXT arranged a trip for two of their engineers, Steve Stone, who was an OS guy, and Trey Matteson, who was just out of Brown, really bright young guy. They both came to visit us in Sandy Hook and read my proposal and were just jazzed, totally jazzed. They're, like, "Oh, my God. This is exactly what we're bumping into. This solves our problems." (p.19)

He asked for something he could carry into the next meeting with his investors.

I said, "Guys, when you go back to Palo Alto I need you to write a letter that identifies what we talked about and why you were supportive of this, so I can basically do the political dance at Stepstone." And they wrote the letter and I have the letter to this day. I think you've seen it. After that they had to let me work on this stuff. So that was how I got to add interfaces and fix some of the runtime problems. (p.19)

The interface declarations he got to add are still in every Objective-C header file today.

Objective-C++ in two weekends, on top of a stranger's open-source work

Engineering · NeXT, around 1990

NeXT's frameworks were written in Objective-C. The companies NeXT most needed as developers, Lotus, Adobe, Pixar, had large codebases in C++. The two languages could not be mixed in one file, so anyone who wanted to use NeXT's frameworks from C++ code had to bridge between them by hand. Steve Naroff ran NeXT's compiler work, and this is how he solved it in his spare time.

The compiler NeXT used was GCC, the free compiler from Richard Stallman's GNU project, which handled C. Naroff had already extended it for Objective-C. The C++ front end for GCC was being written by someone he barely knew.

I briefly touched base with Michael, to get the lowdown on the state of the compiler, and he was pretty optimistic that, though it wasn't finished, it was compiling quite a bit of stuff from the C++ perspective. So, because he gave me enough positive feedback about his work, and, again, I didn't really know Michael, but he seemed really smart, and I trusted him, the little bit I knew him, I decided to try, on a weekend hack, to take his work and add my Objective-C work to it. (Naroff, CHM oral history part 1, p.37)

Michael is Michael Tiemann, who went on to co-found Cygnus, the first company built on free software. The two sets of compiler changes had to be reconciled, because GCC had not been built to be extended by two people at once.

I got Michael's work in a weekend hack, got it pretty far, and was really pretty blown away that, I mean, the grammar was complaining about this, that or the other thing [...] But I basically didn't let it bother me, and I just went heads down and continued. Long story short, after a couple weekend hack fests, because I didn't have time during my normal hours at NeXT to work on C++, got it far enough that I shipped [...] Lotus the compiler, and they were like giddy that, "oh my god, this is like doing a lot of good for us. And please finish this." And that work ended up being used by them, and then the Photoshop team, and Pixar, and it became really popular. (p.37)

What made it possible in two weekends was a design decision about what not to attempt. The ambitious version would have merged the two object models, letting an Objective-C class inherit from a C++ class.

What I wasn't doing, which some people initially thought could be the design point, is to merge the object models, right? Like allow a C++ class to coexist with an Objective-C class. For example, you know, that would allow potentially sub-classing in Objective-C, a C++ class. I thought that was just idiotic, and even if someone was lobbying hard for it, I would just say, "Listen. That's not the design point. It just doesn't make sense." [...] That would be a research project. (p.37)

He kept the two languages separate and let them sit in the same file. Lotus sent him a t-shirt, with a letter he still has.

"As a token of our appreciation for your efforts developing Objective-C++, been authorized to present you on behalf of the Lotus Back Bay team, the coveted code talks, bullshit walks t-shirt." Okay? "The Objective-C++ compiler correctly compiles all our code, allows one to use all the features of both Objective-C and C++ in the same file, and was finished and delivered before we expected it." (p.38)

Naroff on why it worked:

I wish I could say, in retrospect, we were all so brilliant that, you know, all this great stuff worked, because we're such great planners. There was so much serendipity here. We didn't control Michael Tiemann. He came out of the woodwork. And doing what he did is serious work. C++ is a serious language, and it takes a very special person to do what he did. And we leveraged it. (p.38)

Putting Objective-C in the kernel so ten engineers could cover every PC on the market

Engineering · NeXT and Apple, 1993 to 1997

In 1993 NeXT stopped making computers and became a software company selling NeXTSTEP for Intel PCs. That meant supporting hardware NeXT did not control: every network card, every disk controller, from every vendor, each one needing its own driver. Blaine Garst was a NeXT kernel and runtime engineer.

The economics were bad before a line of code was written.

We tried our damnedest to build a profitable product. And at some point the hardware, we couldn't make it on Motorola hardware. We shifted to Intel, tried to be an Intel-based NeXT computer box, but we had to pay the Microsoft tax. Every piece of hardware we sold had to pay. We had to pay 60 bucks to Microsoft because the manufacturers had this anti-competitive [arrangement]. They later were found to be guilty of monopoly practices in this manner. (Garst, CHM oral history, 2:46:59)

Then the driver problem, and the fix.

In order to be viable on an Intel we had to deal with a bazillion different drivers out there. There's a huge space on Intel. And so I put Objective-C into the kernel so that the kernel team could just subclass a new Ethernet driver. Just tweak, you know, with inheritance and just tweak a little bit and new Ethernet driver done. You know, and so that's how a team of 10 kernel engineers could put out a NeXTSTEP that ran on Intel, on many Intels. (2:47:39)

Most network cards from a given family differ from their siblings in a handful of registers. In a language with inheritance, a new driver is the old driver with those differences overridden. Without it, each one is a copy with edits.

Then Apple bought NeXT, and the kernel came with it.

They later shifted that Objective-C driver interface over to C++ when they got to Apple, right? I still, the kernel team still won't tell me who exactly did it. I think I know who did it, but because Objective-C was unknown to anybody at Apple and they just feared it and they wanted C++. So they made a C++ interface, but they change something in the compiler every release. And so now they're using an outdated compiler because it's the only compiler that'll generate [it]. (2:48:13)

The interface that replaced his is the one Mac drivers are still written against.

Prototyping a feature, getting his boss excited about it, then arguing it down himself

Engineering · The NeXT-to-Apple transition, 1996 to 1997

Objective-C sends a message to an object with square brackets: [object doThing]. Every other mainstream language uses a dot: object.doThing(). For developers arriving from C++ or Java the brackets were the first thing they hit and the first thing they complained about, and in the months between Apple announcing it would buy NeXT and the deal closing, Steve Naroff, who ran NeXT's compiler work, built the obvious fix.

It sort of made sense as the new Apple that we might want to start fresh with this new syntax. Because it's more akin to C++ and Java. And I am, I'm not 100 percent certain that I prototyped it in isolation and then showed it to people, or whether someone sort of said, "Why don't you prototype this?" [...] But so I did it, and Avie got excited. And Ali Ozer's [AppKit] team was not thrilled. So perfect example of the tension between, wow, external programmers, that have never seen Smalltalk, aren't familiar with NeXT, but just want standard dot notation. They'll like this work. But the traditional NeXT people are having a fit. (Naroff, CHM oral history part 1, p.65)

Avie Tevanian was his boss and about to run all of Apple's software. Ali Ozer managed AppKit, the framework the syntax would be used against. The people who wanted the change were the developers Apple needed to attract; the people who hated it were the ones who had built everything so far.

Why I ended up being against it was, again, much more pragmatic. I said, "Oh, my god! Objective-C++ has been such a big thing in NeXT world, it's going to be probably just as necessary in the Apple world." And the beauty of Objective-C++ is the Objective-C message expressions are clearly segregated and distinct from the C++ code. [...] If we make Objective-C message sends look like C++, you will not be able to tell the difference. (p.65)

Objective-C++ was his own earlier work: the ability to put both languages in one file. It worked because you could see at a glance which line was which. Dot syntax would erase that.

Now, you could argue, "Oh, well, we're going to have a fancy code sense editor, and you know, the code editor will know, which is true, there's no doubt. The code editor can know, and the language knows. So there's no ambiguity in terms of the language, but just at a superficial level it makes the two languages more integrated, which is why people were excited about it on one level. So I thought keeping them semantically, syntactically and semantically distinct was better than syntactically similar, but semantically distinct. (p.65)

In the end, I sided with Ali's guys and said, "I can't in good faith push this." And I think Avie was very disappointed. And if my only goal in life was to make my manager happy, I would have given in. I would have, and he would have made the resources available. [...] But it's funny, because it just sort of dissipated. It just, and I used to have documents that, for all this stuff. And even that, I abolished from my library. There's no remnant that I could find of any of that. (p.65)

A decade later Apple added dot syntax to Objective-C anyway, for property access, and Naroff says on the same page that he still thinks he was right.

Hiring off the code instead of the resume, then setting the task that made a new compiler arguable

Management · Apple's developer tools, 2005

For twenty years Apple's compiler was GCC, the free compiler from the GNU project. It worked, and it was structured, on purpose, to resist being used as a library: its authors did not want proprietary tools built on top of it. That made everything downstream harder than it needed to be. An editor that understands your code, refactoring, fast incremental builds, good error messages: each one needs a compiler you can call as a component, and GCC would not be one. Steve Naroff, by then running Apple's compiler group, had lived with this since NeXT.

In 2005 an engineer in his group pointed him at a graduate student.

When I looked at Chris's background and what he was doing, I was blown away because, to be honest, I hadn't hired many PhD students because most of them didn't have the hands on the keyboard as much as hands on writing papers. Not to take anything away from people who write papers, but Apple was much more oriented for people that had big ideas with writing code. And Chris's LLVM was open source as well. And so, I was able to look at it. It was much better than looking at the resume. (Naroff, CHM oral history part 2, p.14)

Chris Lattner had written LLVM, a compiler back end designed from the start as a set of libraries, as his PhD work at Illinois. Because it was open, Naroff could read the thing itself rather than a description of it.

Chris had went even further to, I'd say, make himself attractive to me and to a company like Apple where he integrated his LLVM work as a backend to GCC. So, it was like just amazing work on so many levels. And we brought Chris in. And Chris, it was just apparent from the first fifteen minutes that this guy is an amazing developer, designer, person. It didn't take more than fifteen minutes for me to realize we've got to hire this guy. (p.14)

Then what he gave Lattner to do next.

The other non-trivial goal I put on his task is to actually compile the entire system with GCC as the frontend using his LLVM backend, which I don't know how long it took him, but it far exceeded what I thought it would take a mere mortal to do. (p.14)

After he did that, since I'm a frontend guy [...] I said, "You know, Chris, it would be great if we can finally make a compiler, a full frontend, middle, backend, that truly is library-based that is truly going to meet our compile time goals. And while GCC has been great for well over a decade, it's time for a new compiler that will support the IDE." [...] So, I pitched starting Clang. (p.14)

Clang is the compiler every Apple platform has been built with since, and LLVM is now underneath much of the industry's tooling.

"You're not good enough": told he could not present his own work, and calling the car to ask why

Leadership · Apple, 2002 to 2003

Steve Naroff ran Apple's developer tools. In 2002 he took four engineers and some of Jobs's interface designers and spent four months prototyping a redesign of Project Builder, the tool developers wrote Mac software in, so that it looked and behaved like the rest of Mac OS X instead of like the NeXT application it had been. It became Xcode. This is about the meeting where he showed it.

We arranged a meeting with Steve Jobs and Phil Schiller and Ted, and I forget exactly who else was there, but those were the main, Avie was there. That's right. So we had lots of vice presidents in the room, and I gave Steve the demo, and Steve was thrilled, absolutely thrilled. What I thought was going to be a 15-minute demo ended up being a lot longer, because there was a lot of talk in the room. [...] Steve had said, "Well, we have to demo this at my keynote at the developer conference," which was great news. I was like, wow, that's great, and he said, "Who should demo it?" and since I had just given a pretty good demo, I was feeling great. I said, "Well, I'll do it," and he said pretty quickly, and I'm not so sure these are the exact words, but, "Oh no, you're not good enough." (Naroff, CHM oral history part 2, p.4)

I was like, because it's a room of VPs. I had just done this great thing, and he's happy. Well, that was terse, "I'm not good enough." So I shut my mouth, because it wasn't worth getting into it with him there, and he said, "Let's get Chris Espinosa," who was, as you know, with him in the garage, still at Apple today, and Chris worked for me I think at the time working on AppleScript. But Chris was a great presenter, and, yeah, he's very charismatic on stage and just a great speaker. So I said, yeah, cool, so let's ask Chris to do it. Okay. I was quickly past the hurt feeling. (p.4-5)

He was not quickly past it.

The meeting ended. I went back to my office, sulked a little bit like I usually do when that type of thing happens, and I decided I'm going to pick up the phone and call Steve. So I called him, and his, he had someone that was a dispatcher that would find him, so she picked up, and she said, "I'll find him, no problem." So she patched me through. He was driving, and I said, "Steve, why'd you have to do that? I mean, what was that about? You were happy. I just worked my butt off," and he said, "Listen, Steve. I trust you with engineering. You're a great engineer. You've been a great manager over the years. You're not a great presenter," and he said, "It's my stage, and I need to make sure I have the best person there. You've done great stuff. I'm not taking away from that." (p.5)

I said, "Well, that's great to hear, Steve, and I agree with you. I just wish you would've said it like that in the meeting," and he just said, "Well, they all know you're great, and so I didn't have to say it." (p.5)

Naroff, years later:

Looking back, there's no doubt that sometimes I'm sure I was a little thin-skinned, but when you're in the bomb run, the trenches, whatever you call it, and you're working really hard, it's tough to be talked to like that. But what was great about Steve and why we always maintained a great relationship [...] is he understood and, in fact, then obviously patted me on the back and made me feel good. So he was someone that respected that I picked up the phone rather than let it linger or hold a grudge. (p.5)

Chris Espinosa gave the demo at WWDC 2003.

The bug thirty engineers could not find, and the phone that shipped with it

Engineering · The iPhone, 2006 to 2007

A phone has two brains. The application processor runs the operating system and everything you touch. The baseband is a separate processor with its own operating system, running the cellular radio, and in 2007 Apple bought it as a black box from a vendor. The two talk over a serial line. Andy Grignon ran the iPhone's radio engineering, and his account starts with how many other things were new at the same time.

Now we've got everything changing. The apps are different. They have to be custom. We've thrown in fingers as the input mechanism, instead of a mouse, which was actually a pretty big difference. We had never built a keyboard before. [...] So, we had every layer of the stack custom, different, new, full of bugs. And it should come as no surprise that, by the way, with a deadline that was immutable, right? This deadline couldn't change. And Steve was betting the company on it. (Grignon, CHM oral history, p.22)

New chip, designed in-house. Operating system ported to a processor architecture it had never run on. New toolchain. New apps, after the plan to reuse the Mac's had failed. What that does to debugging:

Imagine you're an engineer in this morass, right? And you go to Mail, you check the thing and the phone reboots, which happened all the time. Whose bug is it? Is it the operating system? Is it Mail? Is it the tool chain? Or hey, maybe it's the bug in the silicon, which happened. (p.22)

Several times the whole program stopped because nobody could get past a problem, and the people who had written the software that generated the chip had to be flown in to sit with Apple's hardware engineers. Grignon's explanation for why vendors agreed to that is that they thought they were working on the next iPod. Then the worst one.

One of the worst ones was we had, of all things a problem with what's called a UART, and the UART is the serial block of a chip. It's been around since the dawn of chips. It's the oldest thing ever. It's a serial line, and ours had a problem with it so that the connection between the chip that made phone calls and the main chip that ran all the software in very certain circumstances but not as rare as you'd think, that link would go dead and that would result in effectively a dropped call with the chip that made the phone calls rebooting itself, and so, you now couldn't make a phone call until it came back up. (p.23)

The signal bars went to zero every time it happened. After the phone shipped, Grignon changed his license plate to ZROBARS.

We couldn't figure it out, and you'd have a guaranteed reliable link between these two chips and it's failing. Why? We had rooms full of people, like 30 people who would single step through to see what the hardware is doing. Register by register, it latches this, it does that. Nobody could figure it out. (p.23)

They shipped anyway. The fix was not a fix.

We actually shipped the very first phone with this bug in it, but [...] we took a piece of the Bluetooth stack [...] HCI, something like that. It was a piece of the Bluetooth stack, 'cause Bluetooth radios give you a serial port but it's over a lossy connection, so it's effectively a UART over wireless, which, so Bluetooth solved that problem of a lossy UART, which is supposed to be a hardwired thing. So, we implemented that between our main processor and our phone processor so that when the UART shit the bed that layer kicked in and it kept the link alive and it effectively did a reboot but the chip didn't restart itself. (p.23)

Bluetooth has a layer whose whole job is to keep a serial conversation going over a radio link that drops packets. They took that layer and put it on a wire that was never supposed to drop anything.

Later in the interview Grignon comes back to why it was missed, and the answer is about testing rather than hardware. He had written the bring-up software for the chip, the code that runs before any library exists and asks whether the silicon can add.

You'd be surprised at the things you catch, and we should have caught things in that UART that went sideways. Like, we wrote a bunch of unit tests that test, can this thing move data? Yes, it can move data. We didn't check all of the conditions, and so, that was where we got bit on that. (p.46)

Shooting a process in the head instead of auditing every allocation

Engineering · The iPhone, 2006

The iPhone's software was built from Mac OS X, and Mac OS X assumes it will never run out of memory. On a desktop that is true enough: when physical memory fills, the system pages to disk and slows down. The phone had no paging and a fraction of the memory, so running out was not slow, it was fatal. Richard Williamson, who led the iPhone's Safari and WebKit work, describes what that meant for code inherited from the Mac.

It's pretty clear that we had to do something because the frameworks on OS X are written with the assumption that you have unlimited memory because you have virtual memory. So if you run out of physical memory you swap. And so almost no software is written with malloc, assuming... (Williamson, CHM oral history part 1, p.72)

Ken Kocienda finishes the sentence: "Malloc can't fail." Malloc is the routine every piece of software calls to get memory. On the Mac it always succeeds, so nobody checks whether it did.

Nobody checks whether or not they get a pointer back, whether they got a return value. So we weren't going to rewrite a lot of the software that we imported from OS X onto the iPhone. So it's pretty clear early on we had to do something. (p.72)

The thorough fix was to go through every allocation in every framework and add the check. Nobody had time for that. Williamson's alternative:

The solution was, use a gun and shoot a process in the head if it's misbehaved. So the idea was, in the kernel we were monitoring memory allocations. And if memory levels were getting low or there was an application that was abusing memory we'd just kill it and give precedence to the foremost application. So when you dismiss an application on the phone it doesn't necessary exit. It's given a bunch of opportunities to do things before it has to exit. And if it doesn't exit then we kill it with Jetsam. (p.72)

The name is the nautical term for cargo thrown overboard to save the ship. Rather than make every application handle scarcity, the kernel watches the total and throws applications out, starting with the ones you are not looking at. The application never learns memory was short; it just stops existing.

We met with the kernel guys and pushed this idea forward. And they were like, you know, you're crazy. This is ridiculous. What's going to happen? How are things going to get cleaned up? But eventually they came around. And we implemented it. So it's remarkably seamless, how memory management works on the iPhone. You know, you really don't notice when applications exit and start. (p.72)

Jetsam is still what does this on every iPhone. It is why an app you left an hour ago sometimes reopens from scratch, and why the phone almost never tells you it is out of memory.

The same passage records an argument Williamson lost. He wanted more RAM.

I was a huge advocate for more RAM and the hardware guys went no. And we're talking about pennies. But they insisted on living with the RAM that we had. (p.72)

iMovie's interface shipped as a live Photoshop file, and nobody was told for four versions

Engineering · iMovie, 1999 to 2000

In 1999 there was no pipeline from a design file to a running interface. A designer produced an image; an engineer cut it into pieces, gave each piece a name, and compiled the pieces in. Every visual change, every shade of a button, cost a build. Glenn Reid, who had come from NeXT and was running the small team building iMovie for the iMac, describes what he and a designer did about that.

iMovie, no one knows this except now everyone knows this because I'm being recorded, but we had the same problem. Like we were writing a C program to make movies. We actually had a fixed screen size because it shipped on the iMac, but there were a lot of cooks in the kitchen as to what the interface should look like. (Reid, CHM oral history, 1:35:18)

The designer was Priscilla Shih, hired from Fractal Design.

She and I came up with this scheme. Basically we knew we'd have to keep reprogramming the interface, so we designed a Photoshop file format. We used the layers in Photoshop to represent, for example, a button. You have a mouse-over state and a click state, they change colors a little bit, so those would be four layers in Photoshop. And we made up a little language in the Photoshop layer names, like this is the mouse-over state, this is the click state. We sort of recreated Interface Builder inside Photoshop files, if you will. And then we taught iMovie how to open and read in the Photoshop files to display the interface. (1:35:47)

Interface Builder was NeXT's tool for laying out interfaces visually; Reid had come from the company that made it. The point of the scheme was who could change what.

Because the artists were always changing the bits on us and we would have to keep redoing it. So we were able to give them the Photoshop file and say change it all you want, just don't mess up the layer names, because that's necessary. And we could read the entire, I think there were about 300 layers in the Photoshop file, and we could read the whole thing in and display it in half a second. So it was like a really good way to do user interfaces. (1:36:33)

A development convenience, until it wasn't.

We decided to ship it that way. So built into the first four versions of iMovie was a Photoshop file where we changed the file type so you never knew it was a Photoshop file, and would just read it in and that was your user interface. But if you knew that that was a Photoshop file you could literally open it up in Photoshop, change it and save it again, and iMovie would, you could skin iMovie if you knew that. (1:37:03)

It got better.

In fact you could do it while iMovie was running, because we needed that. So iMovie every two seconds, every one second, would check the file modification time on this Photoshop file, and if it had changed it would dump the UI and reload it. And so we could in real time play around with the user interface while the app was running, and move the button and change the title on it and stuff. It made it much easier for us to do our work, but we didn't really want people to be re-skinning it and screwing around with it, so we just didn't tell anybody. (1:37:28)

Reid says on tape it is the first time he has told the story.

Passing one Safari benchmark with a branch to an absolute address

Engineering · Objective-C garbage collection, Apple, around 2006

Objective-C made programmers manage memory by hand, and getting it wrong was the main source of crashes on the Mac. Garbage collection promised to handle it automatically, at a price: a collector has to watch memory reads and writes, and that watching costs time on every one of them. Blaine Garst, who had built NeXT's runtime and was now at Apple, was the person shipping it.

The gatekeeper was named, and so was the test.

I did a cardinal sin when I introduced garbage collection at Apple. In order to make it fast, I had to prove to Scott Forstall that it wouldn't slow down Safari. (Garst, CHM oral history, 2:55:19)

Scott Forstall ran Mac OS X applications. Safari was the benchmark because it was the application whose speed users noticed most, and a garbage collector's overhead shows up on exactly the kind of tight loops a browser runs. Garst needed the cost of the check to be invisible, and there was one way to make a check that cheap.

The garbage collector intercept point, I broke a cardinal rule. I had the compiler issue a hand-coded instruction to branch to a routine at an absolute address. So how many people get to deploy absolute-addressed entry points in application code? (2:55:38)

Normal code never jumps to a fixed address; the operating system decides where things live, and hard-coding a location is the sort of thing that gets a patch rejected. On the PowerPC there was an instruction that could do it, and it could only reach two places.

The instruction was branch and link absolute, BLA, like my name, Blaine, almost. And it could only branch to the top part of core or the bottom part of core. Bottom part of core, address zero, was taken. So we branched to the high-end address. There was a page or two of possible branch points for that instruction. So we took over the highest page of memory. (2:56:02)

Then the swap.

On the launch of any process we do a little quick check. Are we running garbage collected or not? And if we were, we'd swap out the entry points up there that would go off and do what we call read barriers and write barriers for garbage collection. And if you weren't running garbage collection, it would simply return. So it was a branch and link and return immediate, which was imperceptible in the Safari measurement. So we got to do garbage collection. (2:56:24)

Every memory access in every compiled program got a jump to the top page of memory. For a program not using the collector, the routine there did nothing and came straight back, fast enough that Safari's numbers did not move. For a program that was, the same address held the real barrier code, installed at launch.

The feature shipped in Mac OS X Leopard. Apple deprecated it a few years later in favour of a different approach.

A crash converted into a beep, and a linker fix nobody would make

Engineering · The original Macintosh, 1983 to 1984

The first Macintosh had 128 kilobytes of memory. Applications did not fit in it, so their code was split into segments that were loaded when needed and thrown out when not. Two stories from that constraint, told by the two people who lived closest to it: Bill Atkinson, who wrote QuickDraw and MacPaint, and Andy Hertzfeld, who wrote most of the Toolbox.

Hertzfeld sets the frame, and he does not flatter it.

We were really fighting against a very, very small amount of memory. That's because memory was one of the most expensive components and we wanted our computer to be affordable to ordinary people. So we worked really, really hard to get it to fit in a tighter spot than it really could. And so a lot of the fragility in Mac programs was caused by, we were really shoving more functionality in there than we could fit. (Hertzfeld, CHM MacPaint oral history, p.13)

Atkinson has the number for MacPaint.

I know that MacPaint in its worst case had 134 bytes free. And I could drive it to that state by loading the right big font. And all that and one of the challenges is how do you prove that you've found the worst case use? (Atkinson, p.13)

The answer to that was a tool that came out of something else entirely. Steve Capps had been asked to build a recorder so the team could make tutorials that played the computer back to itself.

So make a mechanism where he could record events and feed them back. So he had the brilliant insight well, you don't have to record the events to feed them back. (Hertzfeld, p.13)

You could make them up. (Atkinson, p.13)

You could just make them up. And so that became a measure of robustness of Macintosh applications is how many minutes or hours could they survive the monkey. And eventually after a lot of work they could last all night. (Hertzfeld, p.14)

The Monkey typed random keys, picked random menu items, and dragged random tools across the screen, forever. Anything that leaked memory on any path would eventually die under it. Atkinson: "MacPaint went two weeks once." (p.14)

That leaves the failure the Monkey could not prevent: fragmentation. Enough memory in total, but no single hole big enough for the next segment.

When code segments were loaded, you needed some code to do this job and that job. They would be loaded sort of at the first available place. But if you needed another code segment and this one would go out, it might leave a hole there that wasn't quite big enough for the next one that you needed but now you had sort of what we called memory fragmentation. That even though you had enough memory total, you couldn't load the pieces of code that you needed. (Atkinson, p.14)

Atkinson's fix did not fix it.

I developed a little technique for this which is setting a flag at the top of the event loop, saying that we failed and as I went to load code segments, if I failed to load one then I would beep and let it go back to the top of the event loop without doing anything. [...] The net result was the user would go to draw something and it would beep and they would try it again and it would work, and they'd shrug and they'd never know that they just avoided crashing the program. (p.14)

A segment fails to load. Instead of crashing, MacPaint beeps and returns to waiting for input. The user tries again, memory has shifted, it works.

The other story is about the machinery that did the loading, and it is in Hertzfeld's separate session, cut from his book as too technical. The segment loader needed a small change to the linker, which ran on the Lisa and belonged to the Lisa team.

I needed some support from the linker, which runs on the Lisa. I didn't have access to the linker source code. Just to make a simple change, just off multiplying something by eight instead of six when it was building something. So, there was no technical limitation. You know, there was nothing hard, but the guy responsible on the Lisa side for doing that hated the Mac, and wouldn't do it [...] I think I never tried to get Steve on it, just because of its technical nature, but I had to come up with a horrible hack that made everything work without them changing it. After the Mac shipped, the guy finally made the change for it, but it was precarious what I was doing. I knew it was, but I got away with it. (Hertzfeld, CHM oral history part 1, p.33)

Moving the menus to the top of the screen, and the second invention that paid for it

Design · Lisa and Macintosh, 1980 to 1983

The mouse was new enough in 1980 that basic questions had no settled answers, including how far the pointer should travel for a given movement of the hand. Bill Atkinson, who wrote QuickDraw and much of the Lisa's interface before moving to the Mac, worked on those questions at home on one of a handful of Lisa prototypes and brought the results in as Polaroids.

I had a Lisa prototype in my home and there were only a few of them, and I would do user interface development at home and then ride my motorcycle in to Apple and show them, show Polaroids to the Lisa team. I took Polaroids and I would bring them in. And so I actually have sort of chronicled in those Polaroids a step-by-step how we bumbled through the user interface for the Lisa which became pretty much what we had on the Mac. (Atkinson, CHM MacPaint oral history, p.4)

One of the steps was where menus live. Every graphical system since has put them at the top of the window, the top of the screen, or both, and the choice is not cosmetic.

At one point I had, I moved the menus from being, I think at that point they were at the top of the window, I moved them to the top of the whole screen so I could always have the full width. If the window was stubby you didn't have to lose any menu titles. And also I'd always have the full height. If the window was down near the bottom, what do you do? Are you going to bounce it upwards or how do you get the menu to show right? (p.4)

Then the effect nobody designed for.

Because the titles were short and the items were long it kind of multiplied your screen real estate by like three times but you sort of didn't realize it. They just appeared. A given menu item was always at the same place on the screen. It was sort of a kinesthetic thing. You could go and move to and start the action before you even sort of, you know, totally aware of what you were doing. (p.4)

Andy Hertzfeld adds the mechanical reason it worked: "Because they were at the very top, you didn't have to aim. It would just, you would just go all the way up." (Hertzfeld, p.4)

The Lisa's application writers did not see it that way.

When I first moved them from the top of the windows and the Lisa application writers screamed. They said "That's going to be so much farther to reach because then we can't, you know, we can't just reach up to the top of the window. We'll have to go all the way to the top of the whole screen." (Atkinson, p.4)

The objection was correct. The fix was a second invention.

I think this is about the time we incorporated variable speed mouse scaling. [...] That if you moved quickly you got a different gear ratio. So millimeters on the table to millimeters on the screen, you'd move more millimeters on the screen if you were moving quickly and that way you'd make sort of a quick upward move and it would move all the way to the top easily and pin there because there wasn't anywhere else to go. [...] It was actually easier than the top of the window because you could overshoot the top of the window. (p.4-5)

And the second invention turned out to be necessary on its own.

That also solved this problem of how do you point between two lowercase "I"s? You need sort of the mouse to go into compound low where a lot of millimeters on the table makes a few millimeters on the screen so you can very carefully point between them. Now I remember the Lisa team saying "Oh this is not going to work. The mouse is going to fall in your lap." You know if you move up quickly and down slowly, eventually it will fall in your lap. And it sort of does but nobody notices. Every so often they pick it up and reposition it and it worked. (p.5)

Hertzfeld, at the demo machine during the interview: turn the scaling off in the control panel and try to use the Mac without it. "I don't think you'll last more than a minute." (p.5)

Stopping work on the Finder to build the database underneath it, against orders

Engineering · The original Macintosh, 1982 to 1984

Bruce Horn came to Apple at 22 from Xerox PARC, where he had spent his teens in the Smalltalk group. He wrote the Finder, the program that shows you your disks and files, and he wrote the Resource Manager, which almost nobody outside Mac programming has heard of and which is why the Finder, and everything else, could exist in 128K.

The Finder started as a demo of something that did not exist.

I had an idea of the files and folders idea. [...] I thought I would do that instead, and the idea of you could open a folder and look at it and close it and so on, I did a demo of that actually from not even looking at the file system, but just looking at a text file that kind of described what a file system might look like. So it read in the text file and you could kind of navigate this. And that was my demo of what I think it should look like. And then I ended up writing the real Finder after that. (Horn, CHM oral history, p.13)

A fake file system in a text file, navigable, shown as "what I think it should look like." Then, on the way to building the real one, he stopped.

Because I kind of think of foundational things, I was thinking, "Well, how am I going to store the icons? How am I going to store the strings?" and so on. And one of the things that I cared about was I wanted to make these programs be able to be done in different languages and not have to rewrite the code, right? So I wanted to separate out all the language stuff from all the code itself. And I started thinking about, well, if I had a little object-oriented database [...] So I ended up going off of my track. And I kind of stopped working on the Finder and working on the fundamental part that would support the Finder and everything else with this little object-oriented database, which was called the Resource Manager. (p.13)

Resources are the parts of a program that are not logic: every string, icon, menu and dialog layout. Keep them separate from the code and you can translate a program to Norwegian, or redraw its icons, without recompiling it. In 1982 that separation was not standard practice, and Horn's version of it had to fit in what was left of the ROM.

I only had three k-bytes of assembly code to work with. That was what was left in the ROM to build an object-oriented database in assembly code. So it worked out, but it was a struggle to make it fit. (p.16)

His manager, Bob Belleville, did not agree that the visible deliverable should wait for its substrate.

Bob and I disagreed on the importance of the Resource Manager. And I just wouldn't yield. I basically said, "No, we have to have this." Right? Because I was counting on it for the Finder. Bob thought, "What's this computer? It's just going to be another computer down the road and another one after that. What's so special about this Mac?" And I thought it was, here was my chance to really do it right, the way I thought it should be done. And so I basically just told him, "I have to do this." And I had backup from Andy and also from Steve to finish the Resource Manager. (p.15)

He got the backing, and he did not make the date.

I do remember at one point promising something about being done in two or three months, and I put stickies up day by day all around my cube, three months' worth. And I'd pull one down every day. And Andy recalls that after the last one was pulled down, I kept working on the Resource Manager anyway. But by then, it was obvious that it was the right thing to do. (p.15)

Why it was obvious: Hertzfeld rewrote much of the Toolbox to use it, so windows and menus and dialogs all lived in resources, and with memory that tight the ability to load and unload objects on demand was not a nicety.

I don't think it would have been possible to do, have the internationalize-able system that we had. I think that we probably wouldn't have been able to do the complex programs that we'd had, because the Resource Manager would allow you to kind of swap in and out objects as you needed them. And we were so constrained with memory that we needed that capability. (p.16)

Then he finished the Finder, with Steve Capps pushing it over the line at the end.

The Finder was 46K bytes. Forty-six K bytes, right and the Lisa Filer that they had done, they kind of redid the Filer after they saw the little pre-demo that we did of the Finder, that I did of the Finder. That weighed in at, like, 360 K bytes. So ours was only 46 K. (p.18)

Why you could name a Mac file anything you liked

Design · The original Macintosh, 1983

Every other system in 1983 identified a file by its name. The letters after the dot told the machine what kind of file it was, and if you renamed it wrong the machine lost track. The Mac did not work that way, and the reason is a design decision Bruce Horn made while building the Finder, which he describes as coming straight out of how he had learned to think at Xerox PARC.

I had this idea of a Type and a Creator. [...] The concept of a Type and a Creator was unique and new to the Mac. Where the Creator would basically be what an application would export as, "This is who I am." And the Type was, the file would say, "This is the kind of thing I am." And you would bind those together. (Horn, CHM oral history, p.13)

Two four-letter codes, stored with the file but not in its name. The Type says what the file is. The Creator says which application made it and should open it. The Finder kept the binding between them in a database of its own, so double-clicking a document knew where to go without asking the filename.

So the Type and Creator system was the thing that allowed you to, for example, name a file anything you like, and all of the type and creator information would be hidden away in the file system that Larry [Kenyon] did. And that was just a bonus. We wanted to name anything you wanted to name and type it with spaces, and we didn't want to have filename suffixes or any of that stuff. We just wanted it to be very natural. And so Type and Creator came out of that. (p.13)

The interviewer asks whether the plan from the start was no extensions, just names. Horn's answer is that the question is backwards.

No, no. These things are objects, right? They have types. They're of a particular kind, right? We want the file to be an object. And so with the file as an object, where do you keep its type, and if it's going to be manipulated by an application, how does it know? And all that had to be part of the system that was built and supported by the Resource Manager, the Type and Creator, and Desktop Database. (p.14)

I grew up thinking object-oriented because I came out of LRG. And I wanted to do as much as I could that would make the system be as dynamic as it was in the Smalltalk world. Of course, you can't do that if you've got a tiny, tiny computer, your operating system is all in assembly code in ROM. How do you make things dynamic? And the best I could do was this object-oriented database. (p.14)

LRG was PARC's Learning Research Group, Alan Kay's Smalltalk team. Horn had been inside it as a teenager.

The design lost, eventually. Mac OS X adopted extensions, because the rest of the world had them and files had to travel.

Choosing the processor so they could keep the graphics library

Engineering · The Macintosh, 1981

The original Macintosh design, under Jef Raskin, used a Motorola 6809, an 8-bit processor cheap enough for a machine meant to sell for under a thousand dollars. The Lisa, Apple's other project, used the 68000, a 16-bit processor with far more power and a price to match. Bill Atkinson was writing QuickDraw, the graphics library, for the Lisa and its 68000. The Mac team's plan was to rewrite it for the 6809. Andy Hertzfeld describes how that plan died.

Bud not only made the decision to use QuickDraw, he made the decision to use the 68000 so we could run QuickDraw. Bud was Bill's best friend. They went to college together. They both went up to Seattle together for grad school. [...] Bud was writing, essentially taking the QuickDraw design, but rewriting it for the 6809 chip that the Macintosh was using at the time, but every evening, seeing what Bill was doing on the 68000 and said, "Boy, I sure wish we could use that. You know, we'd save a year or something like that, if we could do that." (Hertzfeld, CHM oral history part 1, p.19)

Bud Tribble was the Mac's software manager. He was doing the port himself, which is how he knew what it would cost, and he could see the finished version running every evening on a machine he was not allowed to use. The problem was the price.

So he told Burrell, "Hey, isn't there some way we can get the 68000?" Well, we can't for the price point we were aiming at. We could only have one row of RAM. Then Burrell was brilliant, brilliant guy, was inspired, he saw a way to do it, to run the 68000 with only an eight-bit data bus, which was impossible, because it had 16 data lines. But Burrell devised an incredibly clever scheme that he called the bus transformation circuit, which not only allowed him to use the only eight chips, but it ended up being twice as fast as the Lisa design. So, it was both 1/3 the price and twice as fast, this 68000 prototype that Burrell came up with. (p.19)

Burrell Smith designed the Mac's hardware. A 16-bit processor wants 16 memory chips, one row per bit, and the budget allowed eight. His circuit let the processor talk to half as many chips and came out faster than the Lisa, which had all sixteen.

Once we had the 68000, it was obvious we were going to use Bill's QuickDraw as much as we could. I had to make changes to it to adapt it to the Mac's memory management model, which was different than the Lisa's. So there were, I would say, just we used 90 percent of what Bill did, just straight out. (p.19)

The prototype also changed who was interested in the project.

Jef hated it. And as soon as Burrell came up with the 68000 prototype, Steve got very interested in it. Jef was not interested, but Steve was interested in it. (p.19)

The keyboard that lost to layouts that worked better, and why

Design · The iPhone, 2006

Before 2007, typing on a handheld meant a physical keyboard, a stylus, or pressing a number key several times to reach a letter. Nobody had shipped a good keyboard on glass. Bas Ording had designed much of the Mac OS X interface and was now doing the same for the iPhone, and the keyboard was the part he was least sure could be made to work.

We knew it was going to be tricky to get it to work. We had already done some keyboard stuff on the bigger touchscreen. And so definitely I go like, "Oh, now it has to be on an even smaller screen, like this is going to be, this is not going to be easy." So it took us a long time, and like other engineers worked on it. And like come up with ideas, like all kinds of different things just to see, "How can we make something that actually where you can type on it?" (Ording, CHM oral history, p.44)

The interviewer notes there was a contest: keyboard design was opened to the engineers, and everybody built one. Ording confirms it, and says what the best entries had in common.

There was some interesting ideas, or stuff that worked really quite well, where you can't do it wrong really, but then it would be awkward, you'd have to do too much taps or too many swipes or it would just look too weird that you don't recognize it as a normal keyboard. (p.44)

Layouts existed that were more accurate on a small touchscreen than QWERTY. They made errors nearly impossible. They lost.

I think in the end, Steve decided that it needed to be like a QWERTY keyboard, of course, if that can work well, because as you see it, "Oh, yeah! Keyboard!" No big deal, everyone knows what that is. And you know how to type it. You don't have to learn anything. It's like you'd have to be a little more careful, I guess, but then there's like a bunch of smart people that figured out like clever algorithms, how if you type a word, that it makes the best out of it and even if you're a little off, it still corrects it and all that, so it ends up working pretty well. (p.44-45)

Ording, on the state of mind before the correction software worked: "At first we thought, 'I'm not sure how this is going to work at all!'" (p.45)

The three decisions Hertzfeld got wrong, in his own list

Engineering · The original Macintosh, 1984

Most accounts of the original Mac's software are about what its authors got right. Andy Hertzfeld, asked in 2025 about the decisions that hurt the platform later, gives a list, and each item on it was made for the same reason.

The most obvious one, I would say, was running in supervisor mode on the 68000. That was indicative of, get our drive to simplicity. We basically would remove complexity in the model and just make it simpler for the developer, really, if you just didn't need to manage that. That was wrong, because we were going to evolve into a system that needed more protection. We got rid of some of that basic protection, just because we wanted to keep it simple, but that was too simple, I think. (Hertzfeld, CHM oral history part 1, p.23)

Supervisor mode means every program ran with full authority over the machine. There was no wall between an application and the system, so a bug in one could take down everything.

Another bad blunder I made, was using low memory like the Apple II did. I just saw the way the Apple II did the operating system and thought that was good enough and cool. You actually, by putting the global variables in the memory, you save memory space, because the first 32k of memory, the 68000 could address just in a two-byte address instead of a four-byte address. So, I thought that was a good idea, until you considered, well, we wanted to run two programs at once, they both would fight over the memory variable, so that was not a good decision. (p.23)

Another one like that was we added some features, I did myself, to the memory manager. [...] So we put the flag bits to control these new features. Was this object purgeable from memory, etc. We put them in the block header because, well, the 68000 had a 32-bit address, but we were only using 24 bits of that in the model. So, we thought we could use those higher order bits as pointers, simplify things and make things a little faster. That was a disaster a few years later, because we couldn't expand memory past what turned out to be an insufficient limit, and so, we could fix that by changing the system to put those bytes in a different place, but it caused chaos and messed up the early developers, some of them. (p.23)

The processor could address more memory than the machine had, so the spare bits in every pointer were used to store flags. When Macs got enough memory to need those bits, every program that had touched them broke. Hertzfeld had told developers not to. "But people did anyway."

The interviewer offers the summary: decisions that seemed reasonable because the machine had to be simple and cheap, and bit the platform later. Hertzfeld goes further than agreeing.

We didn't know what we were designing for, as history has proven. We thought we were trying to make an exquisite product that's as fast and as inexpensive as possible, but that that would be replaced by an entirely different one within three or four years, just as the Mac was replacing the Apple II, we thought something would come along and replace the Mac. We had no idea that it would last, you know, some of these basic architecture things would last for a decade. So, what we were really doing was designing the first of a long-running platform, but we didn't understand that when we were building it. [...] We hyper-optimized to the machine, where it was in 1984, and it would have been better to make it a little less efficient, a little more memory consumptive, to give it an easier transition into the future. (p.23)

Building overlapping windows because he thought he had seen Xerox do it

Engineering · QuickDraw, 1979 to 1983

Overlapping windows are harder than they look. If one window partly covers another, the machine has to know exactly which pixels of the back window are visible, which means computing an arbitrary shape and clipping every drawing operation to it, fast enough to keep up with typing. The alternative is to redraw everything from back to front each time, which is simple and slow. Bill Atkinson wrote QuickDraw, the graphics library under the Lisa and the Mac, and he solved the hard version for a reason Andy Hertzfeld tells on his behalf.

First, why it mattered. Atkinson on the state of the art:

When you have windows you've got sort of one window and another one obscures parts of it and another one. So when you draw into one of the behind windows you have to clip it to some arbitrary area made by the shapes of the other ones that are overlapping it. And there were ways to draw clipped information of the SIGGRAPH core stuff, you sort of pre-divided the line segments and things like that. But they were only about two orders of magnitude too slow to do a real word processor with. (Atkinson, CHM MacPaint oral history, p.2-3)

A hundred times too slow. Then the famous 1979 visit to Xerox PARC, where Apple's engineers were shown the Alto.

Should tell the funny story about how during the Xerox PARC demo, Apple got one demo from the Xerox stuff, and Bill thought that he saw them drawing into behind windows, windows that weren't the topmost window. And so he knew it could be done and he worked really hard to make it. It turns out that was erroneous. They weren't doing it. He solved, he was motivated to solve the problem because he thought he saw an example but it really was a fundamental problem and they hadn't solved it but Bill did. (Hertzfeld, p.3)

Atkinson's version, including the conversation with PARC afterwards:

Sometimes a little ignorance can be very motivating, and later I think one of the people from PARC said "How did you do that?" "What? I thought you were doing it?" "Oh, no, no. We were drawing the back, we'd have to redraw all the other windows." "Oh, I missed that point." (Atkinson, p.3)

The thing he built is the data structure at the centre of QuickDraw.

So I was able to come up with a very efficient way that really didn't cost much more than drawing into a rectangle but could allow for very arbitrary shaped objects that are clipping. (Atkinson, p.3)

The very center of that in the heart of QuickDraw's capability was the status structure called a "Region" and since you guys are going to have access through the MacPaint source code through this website, you can look at the region code and see how lovingly it was designed and brilliantly it solves that problem. (Hertzfeld, p.3)

Hertzfeld's summary: "It helps to be young to make breakthroughs because you don't know what's impossible." (p.3)

Inventing fuzzing three days before the NeXT launch

Engineering · The NeXT Computer launch, October 1988

The word did not exist yet. Testing software meant writing down the cases you had thought of and running them, which by definition leaves out the bug you have not imagined. Feeding a program random garbage until it breaks was not a recognised technique. Avie Tevanian arrived at it under a deadline.

Tevanian had written the Mach kernel at Carnegie Mellon and joined NeXT to work on the core of its operating system. The NeXT Computer was to be launched in San Francisco in October 1988, and it shipped with a magneto-optical drive instead of a hard disk. Three days out, the driver for that drive did not work reliably.

I have a vivid memory of the lead up, of a part of the lead up to that, which is two or three days before the launch, we were working on the optical drive. And so, I was part of the OS team. [...] I wasn't responsible for the optical driver. I was responsible for more of the core of the OS. And it was working pretty well. But the optical driver was unreliable. And I remember the guy working on it, outstanding engineer. His name is John Seamons. He was burning the midnight oil just trying to get this thing working in about three days before the intro. And this has got to work. I'm like okay, I've got to roll up my sleeves and help John. (Tevanian, CHM oral history part 1, p.47)

The first move was not to look for the bug. It was to build a way of finding out whose bug it was.

I said, "Okay, we've got to do something. We've got to start building reliability tests for the file system, so we can figure out, is it a driver problem, is it the file system, whatever." And so, I built a very simple UNIX level test to read and write files in a slightly random way and ran it. And it failed immediately. And I built in logging and stuff, so we could then figure out what was going on. We went, and we tested. And we looked at logs. We figured out the bug. Great, we fixed it. Good, we're all set to launch. (p.47)

They did not stop there.

Before we do that, let's take the testing to the next level. And so, he and I kept iterating on these test programs to the point where we finally had a test program, which was basically issuing random file system instructions to the OS, completely random, no program would ever do this, just to get to it to fail. And we would see these failure modes. Oh yeah, that's a bug. And oh yeah, that's a bug because even though it's random, it should still work, right? And so, we finally got it all to work. And we, I think the two of us worked almost continuously for forty-eight hours to make that work. (p.47)

Asked whether it was stressful:

I don't remember it being stressful. I just remember being, "Wow, this is so cool." [...] These are the hardest kinds of bugs to find. [...] I remember I learned a lesson from that, which is all of us engineers think we can write code that works. But what you've really got to do to test your code is throw garbage at it and then see what happens. When you throw things at it that is what you expect, yes, it works. But a lot of times, developers throw things at it that you didn't expect. (p.48)

The launch went ahead. The demos, in Tevanian's account, all worked.

Exposé came from noticing what the Dock's animation implied, and its trigger from watching where people parked a button

Design · Mac OS X, around 2003

Bas Ording had been hired to design the Mac OS X interface off a demo, and his tool was Macromedia Director, an animation program in which he wrote small working prototypes. Exposé, the feature that spreads all your windows out so you can see them at once, started as the kind of problem everyone has and nobody had a solution for.

One day I'm just staring at my screen, and I have a bunch of windows and they're overlapping each other. And I'm thinking like, or I was sort of imagining like, "I wish I could just sort of like go behind that window, so I can just see them all." [...] "Well, let's do this, start really simple. What if I just have two windows, and one is above the other one? What would you do to see them both?" I'm like, "Well, I can just like separate them. That would be easy to do. And then if the windows are too big, if they won't fit in the screen, I can just like scale them down." [...] "Oh! well, if it's three windows, then well, it gets a little more complicated, because how are they going to move?" (Ording, CHM oral history, p.11-12)

He worked out an algorithm: shuffle the windows apart iteratively until none overlap, then scale the whole arrangement to fit, and animate straight to the result. A prototype in Director, and no way to run it against real windows. The way in was noticing something about a feature that already existed.

I worked together with John Louch, from Engineering, and he's super good. And he worked on the Dock to get that all to animate, like high frame rates and get it really working really well. And I knew that he could, because of the Genie effect, that was owned sort of by that Dock, the process that runs the Dock and that has to basically grab a window and morph it, right? Or and manipulate it. So I knew that like John Louch knew how to access the windows on the screen. So I'm kind of like, "Hey! If you can grab these windows, you can grab any window, right?" "Yeah, yeah!" "So, okay, well, let's run this algorithm on these rectangles, basically, and see what happens." So we started to do that, and of course, like his code went like ten times faster than my code. (p.12)

The Genie effect is the animation that sucks a window into the Dock when you minimise it. To do that, the Dock had to be able to take hold of any window on screen and distort it. If it could do that to one window on the way to the Dock, it could do it to all of them for any purpose.

So because of John we could get all this stuff to work in the real system, too. And we could just play with it and like really use it for real. And then we could demo to Steve Jobs and a bunch of other people, and they got all excited, and then, of course, you got all the other ideas. "Oh, yeah, it needs to be just the app windows!" Or, "It needs to be all the windows!" (p.12)

Then how you trigger it, which was not designed at all.

For a while we had, this is silly, this like round blue button that we had sitting on the screen that you could drag around anywhere. If you click it, it would just do Exposé. If you click it again, it goes back. But this button was kind of in the way. It was kind of useful, but also in the way. So you end up putting it all the way in the corner, and then we were like, "Well, if we put it in the corner, we may as well just use the hot corners like you use for your screensaver as well, right?" So that's what ended up being one of those features. You just put your mouse in the corner, and all your windows go, "Psht!" (p.12-13)

They shipped a draggable button into the live system, everyone who used it moved it out of the way to a corner, so they deleted the button and kept the corner. Asked whether any of this went through the user testing lab: "No, we weren't. No. [...] For those things, we didn't really do that." (p.13)

The iPod started with a drive nobody wanted, shown at the end of a meeting about something else

Product · The iPod, 2000

Jon Rubinstein ran Apple's hardware engineering, and by 2000 he had been trying for a year to find the parts for a music player. The idea was not the problem. In his telling the storage was.

We needed a small, cheap display. We needed a small, cheap battery. We needed a processor, right, and we needed storage and so I started scrounging around for all this stuff and trying to put the pieces together. And I was hanging out at Almaden at IBM Research because I was friends with Nick Donofrio and Nick said, "Come on by. Take a look at our technology. See if there's anything you want to use." So I took a look and I saw the Microdrive when it was under development and I reached out to the Microdrive Group and said, "Hey, I need five gigabytes and this price." And they laughed at me. Right? "Never going to happen." (Rubinstein, CHM oral history part 1, p.87)

In 2000, storage meant a spinning disk, and disk sizes were set by what laptops needed. A drive too small for a laptop had no market, which is exactly why one was sitting unsold in Japan.

I kept looking and eventually I was in Japan and found the Toshiba drive and they weren't really sure what to do with it because it wasn't, it didn't have enough capacity to really go in a PC. Right? And I was there and they were taking us through their roadmap for all of our other products, and at the end of the meeting, they go, "We have this other thing. Would you like to take a look at it?" I'm like, "Yeah, sure. I'll take a look at it." So we looked at it, and like it's obvious, right? This is how to make an iPod, right? (p.87)

A supplier roadmap meeting for other products, a drive mentioned as an afterthought.

Jeff Williams is there with me, and I'm like, "Jeff, we need to get all of these." So Jeff goes, "Yeah, we think we have an idea. We can use this. How many of them can you make, and can we get exclusive?" And he started the whole exclusive conversation, so I, Steve was in Tokyo, so I said, "Steve," and I've told this story lots of times but, "I need a $10 million check to go do development on this." And Steve goes, "No problem, I'll write you the check." So then I talk to Fred to make sure the check won't bounce because Steve, Fred was the guy who actually wrote the check, not Steve, and Fred goes, "Go ahead. The check won't bounce." (p.87-88)

Jeff Williams, then in operations and now Apple's chief operating officer, opened the exclusivity negotiation on the spot. Fred Anderson was the CFO.

The rest of the parts were already there, for a reason Apple had nothing to do with.

The cellphone industry had really started taking off with, remember, cellphones used to have single-line displays, and it had just been recent at that point in time that the bigger displays, black and white, they were not color, they were black and white, displays had started becoming available, same thing with battery technology. Right? The lithium ion battery technology had just started getting incorporated in cellphones, so there was this convergent of great technologies. Right? And the form-factor was self-evident. It was going to be about the size of a pack of cards because if you took the display and the battery and the circuit board and the hard drive and the connectors you needed on either side, that defined the form factor. (p.88)

One more detail from the same passage, on which the people involved do not agree.

Phil Schiller is the one who came up with the scroll wheel. He had a B&O phone, an old B&O phone that had the scroll wheel, right, and they didn't have acceleration in those days. But, so we grabbed that idea. We patented it right away, and we actually ended up licensing that patent back to B&O. (p.88)

Tony Fadell, who was hired to run the iPod's engineering, gives the same account of the wheel in his own interview and adds a hedge about the years since. On the larger question of whose idea the iPod was, the two men disagree flatly.

Rubber-banding was a debugging aid before it was a design

Design · The iPhone, 2005

Before the iPhone, a list on a screen was moved with a scrollbar, and when it reached the end it stopped. Dragging the content itself with a finger did not exist as a convention, so there was no convention either for what should happen when the content ran out. Bas Ording was building the first scrolling list for the phone, in December 2004 or soon after, and the answer came from a problem with his own prototype.

I remember working on this demo where you scroll through the list and I had lots of times where because I was constantly changing stuff in my code, like, I thought I was running the code, but then it wouldn't scroll and I'm like, "Oh, I guess I'm not running the code yet. Like, wait." But then it was running. But I'm like, "Oh, wait, I'm scrolling in the wrong direction because it's at the top of the list and it's not going to go any further," so I would, it just wouldn't move because it was at the top. (Ording, CHM oral history, p.25)

Two failures looked identical. A prototype that had not started, and a list that had reached the top, both did nothing when you dragged. He could not tell which he was looking at.

And then I started to think, "Oh, maybe I need to add some white space or something that you can see that it's at the top, right." But then still, if you try to move it, and nothing will happen, it feels sort of weird as if the program is just stuck or something. And so that's when I started to think about what can we do to make it feel still alive somehow, that it has some kind of give to it. (p.25)

The fix for his debugging problem was to make the list respond even when it could not move.

One of them was, like, adding this sort of this, yeah, thing where it moves but only at like half the rate of your, that your finger is moving, just to get the feeling of, like, of some kind of resistance and, and then if you let go, it would just spring back to the top of the list. And then from there, like, if you were to scroll at higher speeds and it reaches the end, and it bounces back a little bit and all that. So all the behaviors that go with it, basically to make it feel right. So it took a little while, but it was fun to work on. (p.25)

Every touchscreen since has done it.

I remember, yeah, showing that to Steve and he got all super excited about it, so that was cool. (p.25)

"Interpersonal computing" on machines that could only talk to each other

Marketing · NeXT, 1987 to 1989

Dan'l Lewin ran Apple's higher-education sales, left with Steve Jobs in 1985 as a NeXT co-founder, and ran NeXT's sales and marketing until he quit in 1989. His account of how NeXT positioned itself is the account of the person whose job it was to sell the positioning, and who did not believe it.

The framework came out of a board meeting.

The IBM PC was blue, right? So we had blue, red, and green. And so blue was IBM, and that was all about Lotus 1-2-3 and spreadsheets. And that's what made the PC. And red was Apple, and that was all about desktop publishing. And that's what that was going to be. We were going to leapfrog them all, and we were green. And we were going to be interpersonal computing. So, spreadsheets made the PC, DTP the Mac and Interpersonal computing would be NeXT. (Lewin, CHM oral history, p.25)

Each previous platform had been made by one application category. NeXT's would be communication between people: voice attachments on email, a demo one engineer had built in a week. The problem was the installed base the positioning assumed.

The whole idea was this radical new thing to Steve that people actually were using networks. And we were going to position ourselves as interpersonal computing, when a NeXT machine could only talk to a NeXT machine. I look at it and go, no, this is like a good idea in the shower, but a bad idea in the market. Only our machines can talk to our machines, really? (p.25)

This was 1987 or 1988. Lewin dissented in the room and the positioning went ahead. Then the price. NeXT had been founded to sell a $3,000 computer to students. The 1988 launch priced the cube at $6,500 to universities, and configured it came to around $10,000. Lewin's account of the retreat where the cheaper machine was born says what the gap did to the company.

When Steve came in, and he was sort of morose, and he was down, and I was sitting there, and I sort of said, "you know, we built this company to sell a $3,000 computer. We have a $10,000 computer. Why don't we think about designing a $3,000 computer that people could use and that our market could afford to buy?" Damn, that's a good idea. Let's do that. And so Rich, and George, and Steve, and the whiteboards show up, and all of a sudden it's like, OK, how will we skinny this thing down? (p.30)

Why the price had got there is a design story told as a cost.

The cube itself was supposed to be burdened, fully finished, ready to have the board stuffed in. It was supposed to cost $50. The paint job cost $50. It was a magnesium thing pressed by... you know, it's like... (p.30)

Asked whether he tried to reason with Jobs about it: "Oh, of course! Of course, it was insanity." (p.30)

The go-to-market plan was an attack on Sun's distribution.

Sun had, they were about $400 million, right? They had just gone public. [...] For the most part, they were selling their hardware to an OEM who was bundling software on top and physically delivering the hardware and the software together, whether it was Frame, or Interleaf, or these CAD machines [...] So what we were doing is cherry picking all the software on top, getting those guys to migrate, have Businessland push the distribution out. All they would have to sell is their software. So their cost of sales would change. They wouldn't have to carry inventory, and we could build to order with our factory. So I was going disassemble Sun's entire channel play. (p.30)

Take the software vendors who resold Sun hardware, move them onto NeXT, let a retailer handle the boxes, and Sun's channel collapses. It needed a $3,000 machine. It had a $10,000 one.

We had $120 million or so in the bank, and it was glaringly obvious to me that he was going to burn through all of it in about 12 to 15 months, and there would be no stopping him. The board meeting, I quit the week after this board meeting where Perot blew up and quit. Everybody was done with him. I mean, he had 51 plus percent control of the company, and he was taking no input from anyone. (p.30)

Fifty thousand Macs that dealers would not take, sold to universities instead

Marketing · The Macintosh launch, 1984

The Macintosh shipped in January 1984 into a dealer channel, and Dan'l Lewin's job was the other channel: universities, through what became the Apple University Consortium. It is remembered as a programme to put computers in front of students. Lewin's own account is about inventory.

Because we did the introduction in January, and things sort of frittered along for a while, I was really looking at summer deliveries for fall, for the academic year. And we actually saved, we, our little program, there's a few people in the field that helped me, but it was really my program at corporate [that] saved the company's ass on inventory penalty costs, because the dealer network topped out at about four units a month per dealer. And the factory was running. (Lewin, CHM oral history, p.26)

Four units a month per dealer, against a factory that had been built for a much bigger number.

I think we had about 20,000 units in inventory at launch. And maybe the build rate was about that per month, about 20,000 a month. But it took a bunch of months to get to that point. But then the ramp was going up. And they kept building. And the dealers would only take replenishing their initial shipment, and so all that inventory that got built I took and put into the universities. And so we shipped 50,000 computers in the university community like that. (p.26)

The consortium's big orders were real and were coming: Drexel at 3,000, Dartmouth, and one that arrived as a place in line rather than a delivery.

There was like the purchase order from University of Texas where Charlie Warlick came to me and said I want to get my place in line. And he gave me a PO for 30,000 computers. When do you want those, Charlie? As soon as you can get them to me. You know, and they took a lot of them, but they didn't take 30,000 out of the chute. (p.26)

That period let us turn back the inventory for manufacturing RAM because the penalty clauses and the lead time, especially in those days, was immense. So we both made a lot of money. The margins were extraordinary through that program, but I also saved the company a lot in that period. (p.26)

Memory was ordered months ahead under contracts with penalties for cancelling. A factory building 20,000 machines a month that dealers absorbed at four each was about to trigger those penalties.

The customer who took the Mac apart and told Apple what its network should be

Marketing · Apple and Dartmouth, 1983 to 1984

Before the Mac launched, Dan'l Lewin and Mike Boich were taking prototypes to universities. At Dartmouth, an engineer opened one up.

He looks inside. And Mike and I are sitting there, and he goes, is that an RS-422 chip? And we went, yeah. And so he's, wait a minute. He left the room. A couple minutes later he came back in with the manual for the RS-422 chip. And he's like looking at the manual. And he's in the corner with someone else. And then they go whisper at Bill Arms. And then Bill looks at me and he goes, uh, we got to go for a walk. (Lewin, CHM oral history, p.24)

RS-422 is a serial communication standard, faster than the one in most personal computers of the time. Dartmouth's people had recognised the chip and worked out what it meant before anyone at Apple had told them. On the walk, with the provost, they made an offer.

We're going to tell IBM, who want to give us PC Juniors, and tell Digital, who want to give us DEC Rainbows, to take their gifts and go home. And we recognize you can't give us these computers, because you're just a small company. But can you give us a lab. (p.24)

Lewin had learned one habit from Floyd Kvamme, Apple's marketing head: ask the people who buy from you why they buy, because they will tell you the truth. So he asked.

They said it's very straightforward. And the provost is with us, and he said we're going to make sure that every student has one of these. If they don't have the money, we'll build it into financial aid. [...] Well it's an intelligent graphics terminal that we can put on our network. At which point I said, but there's no networking built into this machine. And they said that's OK. You have a high speed serial chip. We will write our own networking. And I went, and how will you, and they said, through the telephone infrastructure. (p.24)

Dartmouth was the home of BASIC and of time-sharing on mainframes. They did not want a personal computer; they wanted a terminal for the network they already had, and they had found the chip that let them make the Mac into one. Apple's own networking did not exist yet.

We learned that from Dartmouth. And we didn't have AppleTalk, because remember, when Steve got fired, it's partly because AppleTalk didn't work. (p.24)

Then the phone-jack story. Jobs is known to have lectured Bob Metcalfe, the inventor of Ethernet, that nobody would buy networking until the connector was as simple as a phone jack, pulling the phone cord out of the wall to make the point. Lewin heard the story from Bill Krause, 3Com's CEO, years later.

Steve did his Steve thing and got up and walked over, and went over, and "it's got to be like this." And he pulls the RJ-11 jack out of the wall on the phone. It's got be like this. And I just looked at Bill, and I just said, well let me tell you where he got that. Because it was from my trip to Dartmouth where they just said we'll use the phone line. (p.24)

Think Different, as it was actually sold to the room: a schedule and a media buy

Marketing · Apple, September 1997

Think Different is remembered as a piece of film. The tape of Jobs introducing it, eight to ten weeks after coming back, is a different document: a working presentation to Apple's own people that spends its first three minutes on products and inventory before it gets to marketing at all.

We're trying to get back to the basics of great products, great marketing and great distribution. And I think that Apple has pockets of greatness but in some ways has drifted away from doing the basics really well. So we started with the product line. We looked at the product roadmap going out for a few years and we said a lot of this doesn't make sense and it's way too much stuff and there's not enough focus and so we actually got rid of 70 percent of the stuff in the product roadmap. I mean I couldn't figure out the damn product line after a few weeks. (Jobs, Think Different introduction, 0:29)

Then distribution, with a number.

We've got anywhere from two to three months of inventory in our manufacturing supplier pipeline and about an equal amount in our distribution channel pipeline, so we're having to make guesses for five, six months in advance about what the customer wants and we're not smart enough to do that. (2:16)

Only then, marketing, and the argument for what the campaign would not do.

The way to do that is not to talk about speeds and feeds. It's not to talk about MIPS and megahertz. It's not to talk about why we're better than Windows. The dairy industry tried for 20 years to convince you that milk was good for you. It's a lie but they tried anyway and the sales were going like this. And then they tried "got milk" and the sales have gone like this. "Got milk" doesn't even talk about the product. [...] Nike sells a commodity. They sell shoes. And yet when you think of Nike you feel something different than a shoe company. (4:14)

He had fired the agency and the 23-agency review Apple was running, hired Chiat/Day, and given them eight weeks. The film is a minute long and the room saw it once. What the tape preserves that the film does not is the buy.

We are breaking this campaign this Sunday in a rather poetic way. The Wonderful World of Disney is restarting on ABC and the first thing they're showing this Sunday night, I believe it's at 7 o'clock, is the network premiere of Toy Story, and we're gonna have two 60-second spots. This commercial will run twice, once in the first hour and once in the second hour. We are then gonna break some newspaper ads in the Journal, the Times, the Mercury, the Examiner, USA Today, really stating the manifesto, the words. [...] This ad will run throughout most of October on television and we're breaking some phenomenal print within a few weeks, mostly on the back covers of magazines, some on the inside. We've got some incredible billboards and we're even painting some giant walls in about five or six major cities. (10:54)

Toy Story was a Pixar film; Jobs was Pixar's CEO. The premiere was the launch slot. Then the permissions.

In this day and age to use any of these people, whether alive or dead, you need major permission from them, either themselves if they're alive or their estate's representatives if they're dead. Almost all of these people have never appeared in an advertisement before and never would until we asked them. I mean I got permission from Yoko Ono a few days ago to use John. (12:01)

And the close.

Now, advertising is not everything and we've got some incredibly exciting product announcements coming up soon. [...] The question now is not can we turn around Apple, I think that's the booby prize. I think it's can we make Apple really great again. (15:03)

The digital hub was a name marketing put on something that had already grown

Marketing · Apple, 2000 to 2003

The digital hub is taught as a strategy. In January 2001 Jobs stood on stage and said the Mac would be the centre of a person's digital life, the place cameras, music players and camcorders plugged into, and the applications Apple built over the following years are read as the execution of that plan. Glenn Reid ran two of those applications, iMovie and iDVD, and his account from inside is different.

Asked whether the creation of the applications group had been planned:

The creation of the iApps group just happened organically, as you said, and then led to this sort of digital hub strategy that Steve talked about. But it was not planned from the beginning. It was just kind of, it happened semi-obviously by following the media path. (Reid, CHM oral history, 2:39:55)

The media path: photos, then movies, then music, each a thing people were starting to bring home in digital form, each needing something to do with it on a Mac.

Apple was also growing and hiring people and acquiring companies and, you know, there were getting to be a lot of people, and actual marketing people. And so how do we sell this mishmash of weird products? So the marketing people sort of put wrappers around it and called it the digital hub, and the iApps and all those names. They were labels for things that already had organically come into existence. (2:40:23)

They basically came built in. It was a pretty powerful reason to buy a Mac at that point. There was a lot of pretty useful stuff that came with it. (2:41:18)

Boston, August 1997: arguing a booing room out of the position his own company had taught it

Leadership · Macworld Boston, 6 August 1997

Gil Amelio had been removed as Apple's CEO four weeks earlier, no replacement had been named, and Steve Jobs, officially an adviser, was running the company in everything but title. Apple's public identity for fifteen years had been built on Microsoft as the enemy, and the audience at Macworld Boston was made of the people most invested in that. This is the keynote where he announced a Microsoft investment and told them the war was over.

He opened by quoting the press back to itself.

When I started to get involved, a lot of people gave me advice. And some of the advice that was the most popular was Apple has become irrelevant. There was a great one that was Apple can't execute anything. And another one was the Apple culture is anarchy, no one could manage it. You've read all these things in the press. And after four weeks, here's what I found. Quite the opposite of these things, actually. (Jobs, Macworld Boston 1997, 6:49)

Then the conversion of each charge.

Apple is executing wonderfully on many of the wrong things. The ability of the organization to execute is really high, though. [...] They're doing some of the wrong things because the plan has been wrong. And lastly, what I found is rather than anarchy, I found people that can't wait to fall into line behind a good strategy. There just hasn't been one. (7:29)

Then the number, on one slide.

Apple's sales in 1995 were 11.1 billion. In '96, they were 9.5 billion. And in this year, they'll be, you know, 7 billion plus or minus a little bit. That's the problem or the symptom, depending on how you look at it. (8:50)

He replaced the board, keeping two members and adding four. The first boo of the morning arrived on a name.

The first is Larry Ellison, CEO of Oracle. I hope that wasn't a boo I'm hearing. (11:39)

Then market share.

I can't get anyone to tell me the definitive market share number for Apple, but it's around 7 percent from all I can gather. And the question is, where is Apple relevant? [...] It's like 80 percent of the computers used in advertising and graphic arts, design, prepress, all Macintoshes. And 64 percent is the best number I could find. 64 percent of all internet websites are created using a Mac. (18:49)

Who is the largest education company in the world? [...] Apple is the single largest education supplier in the world. [...] Now, I've asked 100 people at Apple this and only two have thought of it. It's incredible the position Apple still has in education. 60 percent of all computers in education are Apples. 64 percent of computers teachers use are Apples. It's a two to two and a half billion dollar business for Apple every year. (20:52)

On the Mac OS.

Most people think that we're about to abandon the Mac OS. [...] Macintosh OS 8, which we just released, was code named Tempo, as you know, right? So, we just released Tempo. Most people think our next release next year, which is code named Allegro, will come out. But then our next release will be Requiem. And it's crazy. (24:13)

Then the deal, in four parts: a patent cross-licence covering everything filed for five years, Microsoft Office on the Mac for five years at the same release cadence as Windows, Internet Explorer as the default browser, and a $150 million investment in non-voting shares locked for three years. The browser was where the room turned.

Apple has decided to make Internet Explorer its default browser on the Macintosh. Since we believe in choice, since we believe in choice, we're going to be shipping other Internet browsers as well on the Macintosh, and the user can of course change their default should they choose to. (28:47)

He restarted the sentence twice over the booing. Bill Gates appeared on the screen behind him by satellite, which the audience booed as well. Then:

Where we are right now is we're shepherding some of the greatest assets in the computer industry. And if we want to move forward and see Apple healthy and prospering again, we have to let go of a few things here. We have to let go of this notion that for Apple to win, Microsoft has to lose. Okay? We have to embrace a notion that for Apple to win, Apple has to do a really good job. And if others are going to help us, that's great. Cuz we need all the help we can get. And if we screw up and we don't do a good job, it's not somebody else's fault. It's our fault. (33:18)

I think if we want Microsoft Office on the Mac, we better treat the company that puts it out with a little bit of gratitude. We'd like their software. So, the era of setting this up as a competition between Apple and Microsoft is over as far as I'm concerned. (34:08)

Then a reframe.

Another bolt of lightning is that Apple plus Microsoft equals 100 percent of the desktop computer market. And so whatever Apple and Microsoft agree to do, it's a standard. (35:18)

The fifteen-point plan delivered uninvited, and the whiteboard binary from years before

Leadership · Apple, 1996

Regis McKenna had shaped Apple's marketing from the Apple II onward and had long since left the company's payroll, but not its orbit. By 1996 everyone in Silicon Valley was having lunch with him about Apple.

A lot of people from Apple and from other places were having lunch with me, or talking to me, and saying, "Something ought to happen at Apple because it's dying." There was just no pizzazz in the company any more in terms of products. The products like their pen-based and voice-based systems, all these were failing. Newton and things like that had failed. [...] I even had people calling wanting to have lunch with me who said that they felt that they could come in and run Apple. (McKenna, CHM oral history part 5, p.7)

He did something about it that nobody had asked for.

I formed an alliance with my firm and Gemini, the consulting group. Some people from Gemini came to me and said, "Something ought to be done at Apple." And so we sat down together and came up with what could they do. We spent an afternoon on a whiteboard doing a session on it, just for our own mental health, I guess. And out of that I wrote a plan as to what they could do in 10 or 15 different steps. (p.8)

Point one was not a product.

One of the first was profitability. I strongly feel that the best positioning tool that any company in the technology business has is being profitable. If you're profitable then the world will look at you and say, "Okay, you're doing something well and right." If you're not profitable they want to know what's wrong, and the CEO gets put on the spot. The company also becomes very vulnerable when you're losing money. It's when your competitors start hiring your people because they figure you're vulnerable. It's when your competitors start going after your customers because they know you're probably not going to be a long-term presence in the market. (p.8)

Then the last point.

Number 15 I remember well. I said, "Bring Steve Jobs back into the company and give him a strategic position of helping to straighten out the company and set his vision." I don't think he thought about that, but that was number 15. (p.8)

He took it to Gil Amelio, Apple's CEO, without an appointment.

I gave him a presentation, probably about two hours in his office, point by point. It involved building alliances with other companies out there and learning from them, that sort of thing. [...] He didn't react much at all to the whole thing. (p.8)

McKenna's read on why:

I think he also relied heavily on his staff. And there was a real arrogance, I think, among his management. I learned later he had developed that at National, we know how to do this thing and nobody else does. And they sort of were, "Go away. Go away. Go away." (p.8)

Its fifteenth point happened a year later, by a different route. From the same session, years earlier: at an Apple staff meeting, McKenna had drawn a fork on a whiteboard.

That's where, in my notebooks, I show the split [in direction] that I had shown them. One was towards IBM. The other was to be a Sony. I said they had to make that kind of decision because it would mean how they staff, how they hire, how they market, everything. [...] I even drew a little triangle, and I put down, "What are the requirements to get into either one?" [...] Sure enough, when Steve came back, he was quoted in Time Magazine as saying, "We want to be the digital Sony." (p.6)

The ledger: promises with dates from the Mac OS X unveiling, and how they aged

Leadership · Macworld San Francisco, January 2000

Keynotes are promises with dates on them, and almost nobody checks. The January 2000 unveiling of Mac OS X and its Aqua interface is checkable, because the developers Jobs brought on stage said specific things about how long their work would take.

The schedule, stated as a rollout.

There's going to be a 12-month rollout of Mac OS X. You can't do these things overnight. It's gonna be a 12 month rollout so we are announcing it today, January 2000. Our developers have already had a few betas of the software. (Jobs, Macworld SF 2000, 1:36)

He then listed the stages: a developer release that month, a beta in the spring, on sale in the summer, preloaded on every Mac a year out. Fifty-five minutes later, closing the segment:

Mac OS X, going to be rolling out on a Macintosh near you the second half of this year. (56:27)

Mac OS X 10.0 went on sale on 24 March 2001. The public beta appeared in September 2000. The three-tier developer story, and its cost estimates, were also stated on stage.

There are three of them in Mac OS X: Classic, Carbon and Cocoa. And the reason for this is to provide a general migration for people from the left to the right over time. [...] Classic runs Mac OS 9 apps as is without modification. We have Carbon which allows the developers to tune up their Mac apps really fast and get all the features of X, and then we have our very advanced object oriented APIs, Cocoa, which lets you write apps in a tenth of the time. (4:30, 22:10)

Then the partners. Adobe first.

Adobe just had an incredible year. We surpassed the billion dollar mark. Steve, if it wasn't for you it wouldn't have happened. Almost half of our revenue, close to 500 million dollars of our revenue, was because of sales of our Macintosh applications. [...] We are absolutely committed to OS X. (Adobe executive on stage, 47:10)

Microsoft committed to two products on day one and left the third open.

Microsoft is fully behind Steve and his OS X product. We're going to be there on day one with Internet Explorer 5 and Outlook Express 5 for OS X. We've got a great new version of Office in the works to ship later this year. We'll go to OS X on that as soon as we can as well. (Kevin Browne, 49:00)

Office for Mac OS X shipped in November 2001. Then two developers, one after the other, with opposite estimates for the same task.

We just got a new system in about a month ago and we decided to try porting our most popular application, which is Flash. 200 million people have Flash Player today. And so we decided to port it, so put an engineer on it, one engineer for a week and a half, and it's running. (Rob Burgess, Macromedia, 51:30)

I wish I could say that ours was only going to take a week and a half. QuarkXPress however is a bit more of an effort. We've been getting a lot of help from Apple and we really appreciate that, but we're going to make it there. (Quark, 52:40)

Flash was a small, self-contained program. QuarkXPress was a decade of page-layout code that the entire publishing industry ran on. QuarkXPress did not ship natively on Mac OS X until 2003.

And one promise from an earlier keynote, with the counterparty standing next to him.

When we announced our partnership with Microsoft two and a half years ago most people couldn't understand it and didn't think anything good would come of it. And I got to tell you, we're working very closely with these guys. Kevin has an awesome team up there of Mac developers and they're doing some great applications on Macintosh. (Jobs, 49:33)

The Boston 1997 bet, closed on stage in 2000 with Microsoft's Mac chief beside him.

Shipping a Unix workstation whose only disk took a third of a second to seek

Product · The NeXT Computer, 1988 to 1989

The NeXT cube shipped with a magneto-optical drive and no hard disk. The drive held 256 megabytes on a removable cartridge, far more than any floppy, and looked like the future of storage. Rich Page ran NeXT's hardware.

The problem was not capacity or reliability. It was time, and what a Unix system does with a disk.

Imagine swapping onto something that has a 20 millisecond seek time versus swapping onto the optical that probably had a seek time of three, four hundred milliseconds. So I mean the optical was good but it was not the best thing to swap to. That was kind of a misuse. (Page, CHM oral history part 1, 2:31:45)

Unix moves memory to and from disk constantly as part of normal operation. Every one of those moves waited a third of a second instead of a fiftieth.

The original cube did not ship with a hard disk. [...] I don't know how long we shipped the optical only but it was like six, nine, twelve months. [...] In the minimum configuration we were shipping it without the hard disk to get the price down, but we came to the realization it's really not very usable without the hard disk. So then that's when we put in the small hard disk. It's sort of an accelerator for swapping. (2:32:10)

Page connects it to the Lisa.

There's a similar problem with the Lisa, right? [...] What happens is you pick a price where the margins are fairly low but livable and you start there and you make the assumption over time you can bring the cost of product down, you know, and maybe hold the price. But the problem is if you have to add anything to the product to make it more usable, that drives the price up and takes time to bring the price of the product down. So those two things work against you. (2:33:29)

Bricking the best iPhones in existence three weeks before the keynote

Engineering · The iPhone launch, December 2006

The phones Steve Jobs would hold on stage at Macworld in January 2007 were hand-carried from China in steel cases, by people who flew there, collected the suitcases, and flew straight back. Each one was graded. Andy Grignon, who ran the iPhone's radio engineering, describes the grading and what his team did to the best of them.

Steve Jobs and Jony Ive would put jeweler's loupes on and they would put gloves on and they would grade the quality of each phone, right, based on the kind of defects it had, because when they were still bringing up a line, plastics don't sit right. They're not meshed or they maybe have a scuff during some bad assembly process, so they're given a grade. So, double As are effectively production level devices. As are really good. Bs, Cs, and Ds fall, like Cs basically just go to QA because they're visually problematic. So, we had I think eight double As and then I forget how many units we had that were As. (Grignon, CHM oral history, p.24)

A new batch had come in around Christmas, and Grignon's team had a software fix to load onto them. They used a gang programmer, a rig that flashes eight or sixteen phones at once.

They go through the programming sequence and they reboot, and all of them as they're coming back up, they crash, crash, crash a bunch of times, and then finally they all go dead. (p.24)

What had happened was inside the radio chip, and it was a feature.

The chip that made the phone calls was the boss, and the security feature in this chip would detect if somebody was tampering with it by squirting their own code onto the chip that would maybe subvert some security policy of the network, maybe make free phone calls [...] After a certain number of iterations, which happened very quickly, if it detects that I've been tampered with, the security feature literally lights a piece of metal on fire inside of itself and it renders it useless. So, it is a fuse that is blown deep in the heart of this chip, and since that chip is dead and since it's the boss, the phone is dead. It's not like you could just lift that chip off, put a new one on. The whole phone's dead. (p.24)

An anti-tamper fuse, designed to stop people stealing phone service, physically burned by a bug in Apple's own update.

We burned up our double As, and some of our As, the best phones that we had. These were supposed to be the phones that Steve is onstage holding that were flawless, the best that we could produce at the time, and we lit them all on fire, well, that little tiny piece of metal. And I thought for sure I was gonna be fired after that. (p.24-25)

Why no test had caught it:

There was a bug in our code that was triggered only when there was an existing software on there already. So, it worked fine if you were just a QA person that had a previous version, you flashed a new build, then it was fine. But if you were taking a brand new piece of thing off of the floor and you put the software down, that's what triggered the security thing, which burned the wire. (p.25)

Every QA phone had been flashed before. The bug only fired on a virgin unit from the factory, which is the one state no QA process ever has a phone in.

You'll see in some of the pictures at Macworld, there's these glass domes and they're really beautiful phones, and they're spinning around and they're off. Well, those are the best ones we had, and since there was high resolution photography, people could get up close to these things, we wanted those still to have that perfect, you know, the best that we could produce. They were supposed to be running a demo loop of just stuff that the phone, like a video that would just play over and over and over again, and a lot of them are off. (p.25)

The photographs from that launch are famous. In some of them the phones in the display cases have black screens.

What it cost the people who built it

Management · The iPhone, 2006 to 2007

Andy Grignon ran the iPhone's radio engineering. This is his account of what the project was like to be inside.

The setting was a room he named.

The people that made the chip, the baseband, were based out of Europe, and we had all these awful bugs in their stack, in our stack and nothing was working right, and it got to the point where we're like, "Look, we just need everybody onsite," and I had them bring like 20 or so people from Denmark and Germany and wherever. We flew them out long term to Cupertino and I shoved all of them in this windowless, cold server room. And we got to name our rooms and I called it European Vacation. (Grignon, CHM oral history, p.32)

A morning bug meeting, every day. One morning after being shouted at by Tony Fadell, who had been shouted at by Jobs:

The project manager sat at the complete opposite end, and we had gotten kind of comfortable with each other [...] he was just kind of leaned back and he had his legs crossed and he was like, "Eh." [...] For whatever reason I just lost it and I just, like a crazy person. I'm slamming my hands on the table, banging, and I just feel myself losing it, right? You know, that's not me, but that was the cauldron, that pressure cooker that we'd been put into. [...] I take my laptop and I threw it against the cinderblock wall [...] In my head it was like an explosion. Like, it hit the wall and parts are flying everywhere, and it was, like, this big, climactic scene, and then I storm out of there. And I'm sure really what happened was it just kind of like, thunk, slid down to the ground, very anticlimactic. (p.32-33)

His own diagnosis: "A lot of us had adopted Steve's temperamental management style, just the explosive, unpredictable, emotional." (p.32)

Then the argument in Scott Forstall's hallway, a week before Christmas.

It was basically like, "Well, if you don't want me to see my kids this weekend at their holiday show then I guess I can handle that now." It was some kind of quip like that, right? And this other person had kids and they're like, it became this surreal debate about who spent less time with their kids. And it turned into this screaming match between the two of them, and this other person storms off so angry, goes into her office and slams the door, just hard, like boom. [...] She slammed the door so hard that the lock mechanism broke and she was unable to open the door handle. (p.33)

The locksmith was an hour away. Forstall, who ran iPhone software, had been working late.

Forstall shows up. He had an aluminum bat and we're all taking full-size whacks at the door handle, at the lock on this door, just like boom, trying to get this thing unstuck. And finally it was Forstall who just went agro on this thing and he beat it to the point where it actually popped off. (p.33)

He follows the stories with this.

I think when you look at a project like iPhone, you know, it takes a pretty significant personal toll, professional toll. I mean obviously it was a successful product, but there was a lot of sacrifices that a lot of people made. (p.33)

At the end of the interview, asked about the ten years since:

What I say is I got a divorce because work is my mistress, right? One thing Steve was really good at was identifying people who will do anything it takes to ship a product, and that works for and against you, right? I'm your person to deliver a new cutting edge product. I will do that at the expense of literally anything else, my own health, my marriage, anything else, relationships. And some of these products, and the iPhone is one of them, requires that, not just from me but lots of people, and that was kind of a pattern when you look at everybody in aggregate, all kind of the same. [...] One of the jokes that we used to say was, the greatest thing about Fridays is it's two more working days until Monday. (p.56-57)

And, in the same breath: "I'm glad it was a success. I wouldn't take it back. I wouldn't do anything really differently." (p.57)

Where these came from

Most of these came from the Computer History Museum's oral histories, three-hour sessions with a few hundred views each, and the people in them are still adding to the record. Two of the sessions here were recorded in late 2025. Everything quoted is verbatim from the transcripts, with a page or timestamp so you can go and check. Where two people remember the same event differently, both versions are here and I have not picked.

If you want the interviews themselves rather than my selection, the companion guide lists every one with its video and what it covers. Start with whichever person you already have a question for.