3.2 The Rise of Massively Parallel Processing (MPP)

In the landscape of high-performance computing during the 1980s, a significant paradigm shift occurred, moving away from the monolithic vector supercomputers, such as those designed by Cray, towards Massively Parallel Processing (MPP). The MPP philosophy posited that computational power could be scaled more effectively by coordinating a vast number of simple, commodity processors rather than engineering a single, increasingly complex and expensive processor.¹ This section examines the rise of MPP through a case study of its most iconic exemplar: the Connection Machine.

Case Study: The Connection Machine

The development of the Connection Machine at Thinking Machines Corporation (TMC), co-founded by W. Daniel “Danny” Hillis in 1983, represented a fundamental break from the prevailing von Neumann architecture.² Hillis’s doctoral research at MIT laid the groundwork for a machine inspired by the parallelism of the human brain, designed to execute a data-parallel programming model.¹

Defining Data Parallelism

The data parallel model is a programming paradigm in which parallelism is achieved by applying the same operation simultaneously to all elements of a large dataset. Instead of a single processor iterating through the data, the model assigns a simple processing element to each data point, allowing for massive parallel execution.³ This approach is particularly well-suited for problems with inherent data regularity, such as image processing, scientific simulations, and neural network training.⁴

To support this model, TMC developed specialized programming languages, including C* and *Lisp, which provided high-level constructs for expressing data-parallel operations, abstracting the architectural complexity from the programmer.¹

Architectural Evolution: The Connection Machine Series

The Connection Machine series underwent a significant architectural evolution, reflecting both the maturation of the MPP concept and the changing economics of the microprocessor market.

Feature	CM-1 (1986)	CM-2 (1987)	CM-5 (1991)
Architecture	SIMD	SIMD	MIMD (with SIMD simulation)
Processors	Up to 65,536 custom 1-bit processors	Up to 65,536 custom 1-bit processors	Up to 2,048 SPARC RISC processors
Floating Point	None	Weitek 3132 FPU per 32 processors	Integrated with SPARC processors
Memory	4 Kbits per processor	64 Kbits per processor	Up to 128 MB per processor
Interconnect	12-dimensional hypercube	12-dimensional hypercube	”Fat Tree” network
Peak Performance	N/A	2.5 GFLOPS (with FPUs)	131 GFLOPS (1024-node system, 1993)

Sources: ¹, ⁵, ³

CM-1 and CM-2: The SIMD Hypercube

The initial models, the CM-1 and CM-2, were the purest embodiment of Hillis’s original vision. Their key architectural features included:

The Connection Machine CM-2, a black cube with arrays of blinking red LEDs. — The iconic ‘cube-of-cubes’ design of the Connection Machine. Image Credit: Computer History Museum

Architecture of CM-1 and CM-2. Image Credit: Greg Faust, Mike Gibson and Sal Valente

Massive Parallelism: A full system contained 65,536 simple, bit-serial processors.
SIMD Execution: A central sequencer broadcast a single instruction to all processors, which executed it in lockstep on their local data. This model was highly efficient for uniform operations across large datasets.
Hypercube Interconnect: The processors were connected in a 12-dimensional hypercube topology. This network provided high-bandwidth, low-latency communication, crucial for algorithms requiring frequent data exchange between processors.⁶
Floating-Point Enhancement (CM-2): The CM-2 augmented the architecture by adding a Weitek 3132 floating-point unit for every 32 processors, significantly improving performance on scientific and engineering calculations.⁵

The physical design, a cube of cubes with blinking LEDs representing processor activity, made the machine’s computational process visible.¹

A diagram showing the complex, multi-layered structure of a 12-dimensional hypercube network. — The 12-dimensional hypercube interconnect of the Connection Machine. Image Credit: Tamiko Thiel

CM-5: The Pivot to MIMD and Commodity Processors

The CM-5 marked a strategic pivot in response to the rapid performance gains of commodity RISC microprocessors.⁷

The Connection Machine CM-5, with its characteristic tall, black cabinets and glowing red LED panels. — The Connection Machine CM-5. Image Credit: MIT CSAIL

Block diagram of CM5. Image Credit: Scott Pakin

Processor node diagram for CM5. Image Credit: Greg Faust, Mike Gibson and Sal Valente

MIMD Architecture: The custom bit-serial processors were replaced with hundreds or thousands of standard Sun SPARC RISC processors. Each processor could execute an independent instruction stream, making the system a true MIMD machine.⁵
“Fat Tree” Network: The hypercube was replaced with a scalable “fat tree” topology, which provided high bisection bandwidth and was better suited to the more varied communication patterns of a MIMD architecture.⁵
Programming Continuity: Despite the underlying MIMD hardware, the system was designed to efficiently support the data-parallel programming model, ensuring software continuity and simplifying the transition for developers.¹

Impact and Legacy

While Thinking Machines Corporation ultimately filed for bankruptcy in 1994, the Connection Machine series had a profound and lasting impact on high-performance computing.

Validation of MPP: The CM series proved that massively parallel architectures could not only compete with but, in some cases, surpass the performance of traditional vector supercomputers. A CM-5 was ranked the world’s fastest computer in 1993.¹
Pioneering Data Parallelism: It popularized the data-parallel programming model, which remains a cornerstone of modern parallel computing, particularly in the context of GPUs.
Influence on Interconnects: The sophisticated hypercube and fat tree networks spurred significant research and development in high-performance interconnects, a critical component of all modern supercomputers.
Scientific Breakthroughs: CM-2 and CM-5 systems were instrumental in advancing research in fields such as quantum chromodynamics, oil reservoir simulation, and molecular dynamics.⁸

The ideas pioneered by TMC laid the conceptual groundwork for subsequent generations of supercomputers and demonstrated that harnessing the power of the many, rather than the few, was a viable and powerful path forward in high-performance computing.

References

Connection Machine - Wikipedia, accessed October 2, 2025, https://en.wikipedia.org/wiki/Connection_Machine ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
Thinking Machines Corporation - Wikipedia, accessed October 2, 2025, https://en.wikipedia.org/wiki/Thinking_Machines_Corporation ↩
Connection Machine® Model CM-2 Technical Summary - Bitsavers.org, accessed October 2, 2025, https://bitsavers.org/pdf/thinkingMachines/CM2/HA87-4_Connection_Machine_Model_CM-2_Technical_Summary_Apr1987.pdf ↩ ↩²
The Connection Machine (CM-2) - An Introduction - Carolyn JC …, accessed October 2, 2025, https://spl.cde.state.co.us/artemis/ucbserials/ucb51110internet/1992/ucb51110615internet.pdf ↩
Connection Machine - Chessprogramming wiki, accessed October 2, 2025, https://www.chessprogramming.org/Connection_Machine ↩ ↩² ↩³ ↩⁴
“The Design of the Connection Machine” - Article in DesignIssues Journal, Tamiko Thiel. Artificial intelligence parallel programming supercomputer design., accessed October 2, 2025, https://www.tamikothiel.com/theory/cm_txts/index.html ↩
Commodity computing - Wikipedia, accessed October 2, 2025, https://en.wikipedia.org/wiki/Commodity_computing ↩
CM-2 | PSC - Pittsburgh Supercomputing Center, accessed October 2, 2025, https://www.psc.edu/resources/cm-2/ ↩