Meta’s Alexis Black Bjorlin (VP of Infrastructure Hardware) discussed AI model scaling, training clusters, and co-packaged optics at the AI Hardware Summit today in Santa Clara. The presentation she gave was interesting not just because of the peek it gave into Meta’s infrastructure but also because of the commentary on future AI systems. In our opinion, this presentation also demonstrated that Nvidia is playing an entirely different ball game than most AI startups. A common trend that we have discussed is the issue of DRAM scaling and network scaling. Both these trends are sides of the same c ..read more

Nvidia has been king of AI training workloads due to their flexible, easy to program, powerful hardware. This may be changing as AI is very dynamic and various different AI workloads are bifurcating. Training isn’t a monolithic entity, and therefore the hardware and software solution best suited for your workload may not be the same as that of another workload. Combined with the rapid pace of model evolution, some AI training hardware is starting to find a niche. Today we are going to dissect the a few of the major players that submit their performance to MLPerf 2.0 as well where this hardware ..read more

Nvidia’s software expertise has propelled them to completely dominating the markets they operate in. This dominance is the most apparent in AI training workloads, but it also extends to the arena of discrete GPUs for PC gaming. Nvidia created an architecture that is highly flexible and programable with the software middle wear on top which offers developers the ability to tap into unheard of levels of performance with nearly the same ease of use as CPUs. This same software then locks users into Nvidia hardware and prevents other firms from getting developer attention for their potentially bett ..read more

This article was originally published on March 23rd via the SemiAnalysis newsletter. A long-time supplier of Nvidia has seemingly been cut out of the supply chain for Nvidia’s next generation GPU. Nvidia unveiled the next generation Hopper GPU for AI and HPC compute applications at GTC on March 22nd 2022. This 80 billion transistor GPU delivers as much as a 6x performance increase in AI Transformer Applications. In most workloads, that performance gain will be 2x to 3x due to newer SMs, tensor cores, higher clock speeds, 4th generation NVLink, and DPX instructions. The top-spec of this GPU com ..read more

Advanced packaging exists on a continuum of cost and throughput vs performance and density. In the first part of the series, we spoke through the need for advanced packaging. Even though the demand for advanced packaging is obvious, there is an incredible number of advanced packaging types and brand names from Intel (EMIB, Foveros, Foveros Omni, Foveros Direct), TSMC (InFO-OS, InFO-LSI, InFO-SOW, InFO-SoIS, CoWoS-S, CoWoS­-R, CoWoS-L, SoIC), Samsung (FOSiP, X-Cube, I-Cube, HBM, DDR/LPDDR DRAM, CIS), ASE (FoCoS, FOEB), Sony (CIS), Micron (HBM), SKHynix (HBM), and YMTC (XStacking). These packagi ..read more

Authors note: Reprinted the title, the original had a crude analogy related to drugs and addiction embedded throughout. I have donated $1k to the opioid crisis, I recognize that this doesn’t absolve me in any way. People all over the semiconductor world have been speculating about the TSMC and Intel deal. All sorts of rumors have circulated about this deal from Intel trying to stiff AMD out of capacity to Intel running with their tail between their legs begging TSMC for capacity because their new nodes don’t work properly. This has all culminated with Pat Gelsinger’s trip to Taiwan last week ..read more

Advanced packaging has been an increasingly common theme with semiconductors over the last handful of years. In this multi-part series, SemiAnalysis will break down the mega-trend. We will do a deep dive into the technologies that enable advanced packaging such as high accuracy flip chip, thermocompression bonding (TCB), and various types of hybrid bonding (HB). This deep dive will include the status of usage, equipment purchasing, and differences in technology choices for various foundries, IDMs, OSATs, and fabless design firms. It will also contain commentary on the equipment and supply chai ..read more

Tesla hosted their AI Day and revealed the innerworkings of their software and hardware infrastructure. Part of this reveal was the previously teased Dojo AI training chip. Tesla claims their D1 Dojo chip has a GPU level compute, CPU level flexibility, with networking switch IO. A few weeks ago, we speculated on the packaging of this system being a TSMC Integrated Fan Out System on Wafer (InFO_SoW). We explained the benefits of this type of packaging alongside the cooling and power consumption involved with this huge scale up training chip. Additionally, we estimated that this package would ou ..read more

Nvidia is the backbone of the Artificial Intelligence revolution.               Nvidia is the only firm providing vertically integrated artificial intelligence solutions for the datacenter, edge, robotics, healthcare, the digital metaverse, and manufacturing. Nvidia is the only semiconductor firm that actually understands software. Nvidia is the only firm that is cloud native, but understands its biggest competitors are the cloud giants. Nvidia is the only semiconductor firm that is fighting against these cloud giants by all ..read more

Recently MLPerf v1.0, an industry standard on measuring AI performance in standard models, was released. MLPerf Training measures the time it takes to train machine learning models to a standard quality target in a variety of tasks including image classification, object detection, NLP, recommendation, and reinforcement learning. Processors from the likes of Nvidia, Google, Graphcore, AMD, Intel, Habana, and Huawei all made appearances from a variety of server OEMs. Nvidia offered the widest array of results from 4 GPUs all the way through 4096. Google also had some extraordinarily strong showi ..read more