OpenAI revealed its Jalapeño inference ASIC at Hot Chips in August 2026, a chip that leaned heavily on AI to deliver an incredibly short design window. Following the reveal, Tom’s Hardware had the opportunity to sit down with the company’s VP of Hardware, Richard Ho, to answer some of our most pressing questions about how the chip came into existence, future ambitions, and how AI might be used in the development of silicon.
The following article is a full transcript of our interview with Richard Ho, which has been edited for flow and clarity. You can also read additional interview transcripts we produced earlier in the year, featuring Intel, AMD, Nvidia, Valve, and more. This transcript is free to access for a limited time as part of Tom’s Hardware Premium’s AI Chip Design Week.
Jake Roach, Senior CPU Analyst, Tom’s Hardware: It was quite the ending to Hot Chips when you dropped this. I want to start at a high level. There are a lot of reasons for OpenAI to develop its own ASIC, but was there one thing that you could point to more specifically that was a driving force? Whether it’s performance, efficiency — what was it that really drove that decision?
Latest Videos FromTom’s Hardware
Richard Ho, VP Hardware, OpenAI: It is efficiency. I think that’s the main thing that we’re aiming for, because obviously, as Sam [Altman] has been saying, we are going to be compute-limited, and a compute limitation is really how much power we can get into data centers.
What we want to do is be as efficient as we can with the limited compute and limited power that we’re going to be able to get, and make the most of it. Because what we really care about is how much intelligence we can deliver to the users, and having a more efficient inference device is very useful. That’s why we focus on inference, because training happens, and you do a lot of compute with the pre-training, but really the cost to the user is on the inference side, and their perception of intelligence is going to be there. Their user experience in terms of how fast ChatGPT responds, or how fast Codex responds, or how fast the agents respond — the latency really matters.
You can see all of those things in the ingredients of what we announced. You can see that we have both a very good low-latency device for those who really care about it, and we can very easily just turn the knob and get very good throughput, so you can reduce the cost of that inference. Really, that’s the thing that we were aiming for. I’m very happy that the team managed to deliver that.
The benefits of building in-house
Roach: Developing your own ASIC versus going with something that’s currently on the market— were you just not satisfied with the efficiency of current offerings?
Ho: Well, I wouldn’t say that. The way to really think about it is we wanted to take advantage of the co-design opportunity that we had. It’s something that you can’t do with a third-party silicon merchant really well, because there’s a lot of research IP in the models. You just can’t share that widely because it will leak. It will get out there no matter how many NDA’s you put in place.
Ho: Having an internal team being able to work with our researchers, who are able to have full visibility into the full stack, and take care of that: “Should we do this in the software? Should we do it in the model? Or should we do this optimization in the hardware?” We can make those trade-offs intelligently because we have that full visibility, and I think that’s where it comes from.
A lot of the benefit of Jalapeño is that visibility that we had and [we were] able to see exactly what hardware was needed to make those trade-offs. Be intelligent, put whatever we need to put back into the compiler, back into the stack, but really make the hardware fly for this particular application here, which is the outcome of this full-stack co-design opportunity of being inside OpenAI.
Roach: I did want to clarify some points here, because there’s been various quotes floating around about optimizing for this specific workload, and I think that has been maybe misattributed to optimizing specifically for OpenAI’s workloads.
Ho: Yeah, it’s misattributed. The whole point of using the InferenceX benchmark from SemiAnalysis was that it was (using) open-source models, and they’re different models. The architecture is different, and their sizes are different. What we really wanted to demonstrate, to put to rest, the misperception in the industry that our custom inference chip was only for OpenAI models — we’ve shown with the Hot Chips results that it flies on open-source models, flies on any LLM, in a sense. All transformer-based LLM models will be very performant.
The thing we also wanted to show was just how easy it was to program. Taking these models, which we did not even look at until after we got the chip back, and getting them up and being performant in two months, roughly, and being able to present the results, shows it’s programmable, it’s general-purpose, and it’s not hard-coded for OpenAI models.
Roach: It leads me to wonder: Jalapeño is obviously for OpenAI’s inference workloads. Is that all it’s for, or are you considering external customers? What is the plan with OpenAI hardware?
Ho: You could use it for anybody, honestly. But we have such a strong demand for compute within the company. It’s going to take us a good long time to even fill our own demand, which is growing all the time.
With the growth of the daily active users and weekly active users, with the new models, with the new capabilities of Codex, and all the other reasoning things that are going on — and there’s new announcements coming that [are] not out yet, but we kind of know internally — I think that we’re going to have our hands full just providing compute for OpenAI for a good long time. That’s not to say that it can’t be used elsewhere. I believe it could be, but I think our priority is to make sure that OpenAI’s compute needs are met first and foremost.
Roach: Moving to the timeline, it’s remarkable that — what, nine months, I think it was, to initial RDL the tape-out? Really remarkable. Assisted by AI. Does that get faster? Is this kind of ground zero of what we can do with an AI-assisted design process? Are you able to move quicker as you ramp up your roadmap?
Ho: The way I like to think about it is, we’ve established a new baseline. In the old baseline, you’re talking 18 months to two years, roughly. Often that’s even with some existing IP or some more legacy architecture design. We’re starting from scratch here. We had nothing. There’s not a line of code here to refer to.
What we’ve established is that there’s a new baseline that you can do with a very talented team with the help of AI. Now, does it get shorter? It depends on what you’re trying to do.
With Jalapeño, we made some, I consider to be, smart and pragmatic trade-offs on the architecture, the microarchitecture, to hit a very fast time to market because the compute need was so high. It’s like, “Okay, how fast can you get this device for us?” There were some pragmatic trade-offs.
If you were to make a much more complex device — and technology is coming along, with 3D stacking, with co-packaged optics, and stuff like that — will it take longer? Will it take nine months? I won’t say it will take nine months. I think it will go faster than if you didn’t have AI models. If you’re doing a derivative design of Jalapeño, it should go much faster than that. We should be able to do that really, really fast.
What we’re saying is that I think we’re establishing a new baseline: nine months from scratch. Then you’re going to have your usual engineering ups and downs from there. But we think that every engineering team in chip design should be able to use this as a new baseline, because it’s a proof point that the models that are in use — the AI models for us is mostly Codex, Sol, the one before Sol, and now we’re moving on to Astra. These are super capable.
Even from when we started that work, back in November 2025, to when we taped out, the models improved enormously. Even from that moment to when we started doing the kernel optimization in May, when the chips were first coming online, we ourselves were shocked at how much better Codex was and what it could do.
I’ll be honest with you: We were actually a little bit surprised at the performance we were able to squeeze out in those two months of sprinting on the benchmark, because we just didn’t realize just how good the models were at doing kernel optimization.
That’s something that everybody can learn from, to be honest. It’s a proof point that it can be done. This is how AI should be used. We didn’t replace our engineers; they just became super productive. With a smaller team of really good engineers with a lot of this AI stuff, you could do things faster and better than you could otherwise. I think that’s a good model of how engineering should be approached in the AI age.
Roach: I appreciate that insight. I know for at least some of our readers at Tom’s Hardware, the idea is, “Make me a CPU” in ChatGPT, and then it spits something out. But obviously, a lot more has gone on.
During the development process, are you using standard EDA tools from Cadence and Synopsys? And where are those?
Ho: I think this is super important. In general, my team is very open-source-pilled in many ways. We actually put stuff back into open source, and we were open-source-pilled before we got here.
But for sign-off, you need to use the standard EDA flows, and we did, because you want to make sure those results are good and correct. There’s no real alternative today. Part of it is this combination of standard flows optimized with AI, optimized by really good engineers.
Interest from the wider industry
Roach: Obviously, you guys work with hardware vendors across the industry. I’m curious if you’ve had conversations with them post-Jalapeño reveal about this AI-assisted process, and if you’ve heard anything from them.
Ho: Yeah, we engaged with them before the reveal as well because we knew the results were there, so we started talking with some of them. Post-review, we did get a lot more communication with them.
I’m not going to preempt anything here. I can tell you that there’s a lot of interest in the industry, and I would also tell you that we feel that there is a lot of benefit in industry generally that we want to enable.
This is not something that, “Hey, we have this, and we’re going to keep it to ourselves.” It’s not one of those things. We want to make the industry more productive in general because better compute from everybody helps us as well, and so we want to make sure everyone gets it. I think that’s something that we’ll see more about quite shortly, to be honest.
Roach: Just to clarify, when you’re saying you’re seeing interest from the industry, that is for the design flow, how you built the chip, not necessarily, “Hey, we’re going to throw out a bunch of Jalapeños to everyone.”
Ho: Right, exactly. What we did to make those timelines, what we did to get the performance boost at the end. How did we do it? What did we use? I think those are learnings that we want to bring out to the industry as well.
As you probably are aware, there is a pretty active startup scene around AI for chip design, and it’s good. There’s a lot of smart people thinking about it and trying to do it. We have our take on that, and I think at some point we want to tell the world, “Here’s our take on it.”
Fundamental to that is Codex and GPT-6 Astra coming out. Those are fundamental, and we can basically point to it; it’s not going to be slideware or vaporware. We can point to it and say, “Here’s what we did, here’s how we did it, and here’s what we got.” It’s going to be very concrete.
Roach: So it was a proof of concept that ended up being quite a bit faster than expected?
Ho: Yeah. To be honest, the way it worked — and I’ll give you a little bit of insight — our engineers were just like, “Oh, we have these models, and they’re kind of cool. Should we try them?”
They tried them. There was no real “This is what we’re gonna do, and here’s how we’re gonna do it.” They just tried it at a grassroots level. Then they’re like, “Oh my God, it’s so good. Hey, come over here, have a look at this.” Then slowly the whole team got, “Oh man, this is really useful and really good, and here’s how we’re gonna do it.”
The researchers helped us. The researchers we have in the company helped us when we ran into some, “Oh, it doesn’t quite do it this way,” and we’d ask them, “Is there any way we can fine-tune it or something like that?” Then they would come back with replies.
It was really a collab between the chip team and the research team. We do sit with them. The chip team is actually considered part of that research-adjacent organization here within OpenAI. The collab has been really close, because that’s how we got the co-design to start with.
But then this part was almost a bonus. We didn’t set out necessarily to do this as a target for what we did. It just turned out that, “Oh yeah, this is really useful.” The engineers loved it, and now we have a way to do this.
Roach: I want to zoom out a little bit here, because one of the concerns with everyone right now is supply, just in general. Not only supply, but even space to do anything. Where are you with that? I’m assuming you’ve anticipated this situation and have secured your supply.
Ho: It’s a hard situation because supply is very limited. I don’t want to say, “I told you so,” but two years ago, Sam (Altman) and I were doing a tour around all the different fabs and suppliers. I went to say, “Please, please, please build more, build more. We’re gonna need it.” And they were like, “Hey, trust us. We’ve seen the cycle before.”
But I think it’s now become evident to everybody that, just like we were talking about, there’s a new baseline for how to do chip design. There’s a new baseline for supply and what we need in terms of memory, in terms of logic wafers, in terms of all the rest of the components that go into it, SSDs and everything else. The supply chain is responding, but it takes years to get that going.
We’ve seen this for a while, and so we’ve been active in trying to make sure that we have our supplies established and set up. We think we are in good shape for that.
Using Turing over Vera, and OpenAI’s north star
Roach: I cover chips broadly at Tom’s Hardware, primarily focused on CPUs. We have someone who focuses more on graphics, and it was interesting to me to see — I was reading the SemiAnalysis article about it, about the Turing rack that goes alongside a Jalapeño rack.
What was the decision there for Turing and not Vera? I would have expected, given the close working relationship between Nvidia and OpenAI over the years, I would have expected Vera. What makes Turing the right fit?
Ho: The way we approached that design was really in terms of de-risking and being able to do that design fast. Vera, as a standalone, is a little bit behind on that maturity level. The Turing device is strong. It did what we needed to do, and partly our partners had some experience with it.
It wasn’t necessary for us to take a huge risk on that, and so we didn’t. As I said earlier, for the Jalapeño program, we were trying to make very pragmatic decisions. We wanted to be aggressive on the goals of the performance and the cost, but we didn’t want to take unnecessary risks. That felt like a good design decision that would fit within the parameters of how we make these design decisions.
Roach: I’m curious internally: Is the scope of Jalapeño right now — you have a huge compute need. It may not even be able to satiate that. Would the idea be, “Hey, if we can run everything on our own accelerators one day, that’s great”? Is that the ultimate pie-in-the-sky goal?
Ho: I think the ultimate goal is to use the best device in terms of performance and cost. If it turns out that it’s our own internal device because we can do the co-design, because we can do the rest of it, and we then get the performance benefit per watt, then yeah, let that be the case.
But if it’s not, if there is another chip that’s provided by a silicon merchant or another partner, I’m more than happy to put those in the fleet. Our goal is to lower the cost of infrastructure. That’s what our goal is. Whatever the best way to do it is, we’ll do it.
Now, we’ve taken a bet that we can do better than merchant silicon because of this co-design benefit, and it seems to be paying off with Jalapeño. Will it continue paying off? I believe so. But am I going to say that’s our north star? No. I’m going to say our north star is the lowest cost of infrastructure we can get.
Roach: I wanted to ask you the question because I know we’re going to get comments about, “Oh, they’re still using Nvidia. They’re still using AMD.” Obviously, you guys use everything at this point.
Ho: We use everything at this point. But as you know, it’s a constant — you can’t imagine — it’s a constant evaluation. We’ll constantly be evaluating as we continue to deploy, and so the ratios might change.
But as long as we keep that North Star in mind, what is the best device, and not have this attitude of, “Well, we built it, so it has to be there” — and that’s not the way we think about it — then I think we’ll be doing the right thing for both ourselves and our end customers.
Speculative decode and performance on Jalapeño
Roach: Drilling down a bit more into the technical weeds — I know we’re running up close here — speculative decode is currently not implemented on Jalapeño. Do you plan to implement it on Jalapeño?
Ho: It’s implemented in the hardware. The only reason we didn’t benchmark it is that we didn’t have the time to train those draft models to do the speculative decode with Jalapeño. We have internal models that do it. That’s the only reason we did that benchmark again.
Just to give you the background on that one. We did not plan on doing this benchmark until after we got the chip back. We saw it was working; it was working really well. They said, “How are we gonna tell the world about this?” And we said, “Oh, that benchmark. Let’s go for it.”
It was crazy. It was two months until the paper deadline — the presentation deadline. Just like, “Can we do that? And how many of these models can we get done?” We just went for it, and we said, “Well, there’s no time to do the multi-token prediction, but hey, it looks like our single-token prediction might be better than the multi-token prediction. So let’s just publish those results, because that gives you an indication.”
The multi-token prediction is going to be 3-5x performance. That’s in our pocket. We have that available, and we’re going to roll that out in our own models. When we put it into production, we’re going to have that available.
But it didn’t seem necessary to do that for the benchmark, because if your single-token prediction is better than the current state-of-the-art multi-token prediction, you know that your multi-token prediction is going to be much better. That was the reason behind that.
Roach: So you’re saying you get the chip back. You have two months. You weren’t even planning to show off benchmarks originally?
Ho: No. The Hot Chips organizers were actually very flexible and nice. They asked us, “Do you want to present this year?” And we said, “We’re not sure. How late can we tell you?”
They had this last slot, and they kept it there. And if not, the program would have just ended earlier or something like that. Then finally, very late, after we got the chip back, we were like, “Can we have it?” And they said, “Yeah, go for it.” And we went for it.
Roach: Another question I had is looking at longer context windows. If I’m not mistaken, all of them are 8K/1K on the SemiAnalysis InferenceX benchmark. I know you haven’t shared those. I’m assuming you’ve looked at longer context windows internally.
Ho: Yeah, internally, of course. Like we said, I think we said this somewhere in one of the slides, is that on internal models, the gap gets even wider. Yes, with longer context, with even bigger models, Jalapeño seems to perform even better than the existing benchmark from some of the other devices that are available.
I would also highlight that we did a lot of comparisons against Grace Blackwell, but that’s because those were the best published results that we could find. Obviously, by the time we deploy, it’ll be Vera Rubin, maybe even Vera Rubin Ultra in some parts of the deployment schedule.
We’ve done our internal ones, but obviously we don’t publish those. Those have to come from Nvidia and other people who are able to do that. We won’t publish those results ourselves.
Ramping and the future
Roach: I believe this is right — Jalapeno has a slow ramp-up through the rest of the year, but 2027 is when the ramp really…
Ho: 2027. Yeah. I think we want to get some amount in there if we can, a very small volume, just to make sure that everything’s working well and we can test in the production environment. Then 2027 is when the ramp is going to really show up.
Roach: Going back to your roadmap here, you wanted to set a new baseline. You’ve obviously announced two more generations […]Gen 2 is approaching tape-out. Is that the same cadence moving forward? Is it around Hot Chips next year when we should expect to learn more about Jalapeño 2?
Ho: Let me be clear about that. I’m not of the opinion that you should just tape out on a calendar schedule.
We want to tape out when the device that we have in mind makes some kind of step-function improvement in some way, like performance per watt, raw or latency. A lot of that is dependent on when technology becomes available.
Whether it be which generation of HBM you’re using, which type of SerDes you’re using, or whether you can get optical communication closer to the silicon. Our projects and tape-outs will be dependent on the maturity level of the technologies.
We can do very fast execution. Will we do those types of executions back-to-back? I doubt it, because the technology will not be ready for that, and I don’t want to tape out something that is 2% better than what I taped out before, because it’s not worth it to change a fleet. But what I’ve been seeing is the technology does improve at a cadence which is pretty reasonable, and we’ll be able to do our execution fast within that.
The most important thing is building the right device. You’ve got to spend enough time to build the right device. Know what that device is. Then when you build it, just build it fast. Get it out as fast as you can.
But you need to spend enough time to know what’s the next best device, and it’s not just a routine on the treadmill type of thing. That’s the thing I don’t think we want to be on the treadmill for, just for the sake of it. We want to really make a step improvement with every device we do.
Roach: One of the most interesting slides to me is really early in the presentation, where you’re listing out goals, and not only goals, but you listed out the non-goals. I thought that was really telling, because you got to define what you’re not trying to do.
Ho: Exactly. We want to be very thoughtful about our program here because we are a very small team. The things that we do, we want to make a really high impact, and the impact is for the north star: enabling more intelligence, more cheaply for our customers. That’s the thing we’re trying to do.
We want to be very thoughtful about what it would take to do that. As I said, if something can be provided by the ecosystem, and we can’t do better than that, then we just take the ecosystem.
We’re going to always do something that we think is taking advantage of our co-design, taking advantage of our knowledge of where things are going, and being able to use that intelligently.
[Session Ends]
