Hardware of the InfiniBand Cluster


Hardware InfiniBand-Cluster



36 Opteron nodes allow the execution of 64-bit and 32-bit applications. The nodes have four AMD Opteron 850 processors at 2.4 GHz with 8 GB of main memory. opt25, opt26, opt27 have 16 GB of main memory. Each Opteron node has two local SCSI disks running in a RAID 1 (mirroring). The nodes are equipped with two Gigabit Ethernet ports and with a MT23108 InfiniBand Host Channel Adapter card from Mellanox.


Four Itanium2 Nodes allow the execution of 64-bit applications. Each of the nodes has four Intel Itanium2 processors at 1.3 GHz that share 8 GBytes of main memory. The chipset on the boards is an Intel 8870. Each Itanium2 Node has a fast local SCSI disk for booting and local file storage, and a Gigabit Ethernet port. A MTPB23108 Host Channel Adapter card from Mellanox Technologies uses a 133 MHz PCI-X slot of the main board and provides two 4X InfiniBand ports. The four Itanium2 Nodes (and an additional 8-port InfiniBand switch) are a donation from Intel.

An InfiniBand Switch MTEK43132 from Mellanox Technologies is the core of the InfiniBand fabric. The switch has 96 ports, all InfiniBand connections are 4X with a theoretical peak bandwidth of 10 Gbps in each direction.

Two file server with a RAID provide a common home directory and other common files. The raw capacity of both file servers is 3.4 TB. At the moment only one file-server is used. A dedicated login node serves as a gateway to the University. All nodes are connected through a private Gigabit Ethernet

Some pictures of the cluster can be found here.


 Itanium2 Nodes Opteron Nodes
Number of nodes
 4
 36
Processor
 Intel Itanium2 1.3 GHz
 AMD Opteron 850 2.4 GHz
Number of Processors
per node
 4
 4
L1 data cache size
(line size)
 16 KB
(64 bytes)
 64 KB
(64 byte)
L2 cache size
(line size)
 256 Kbyte
(128 bytes)		 1 MB
(64 byte)
L3 cache size
(line size)		 3 Mbyte
(128 bytes)		 N/A
Main memory
 8 GB
 8 GB (some 16 GB)
Disk subsystem
 single SCSI disk, 73 GB
 2x 73 GB SCSI
Network
 4X INFINIBAND & Gigabit Ethernet 4X INFINIBAND & Gigabit Ethernet