8 GPU watercooled computation rig using Power Color Devil 13 HD 7990 cards

Workstation motherboards provide 4 or more PCIE x16 gen 3 slots with a good distribution of dedicated links direct to multiple onboard CPUs. With dual cards such as 7990 which are in fact 2×7970 each one can in theory load up a 4 slot motherboard with 8 gpus in total. Alas, in practice, there can be obstacles booting with 7 or 8 of them with certain motherboads as they tend to run out of PCIE resources after around 6 cards which could be considered to be surprising for a workstation class motherboard. Incidentally, I think it’s worth noting that not only are these the only model of 7990 made but they are practically impossible to get hold of so very rare gems indeed.

Watercooling four dual gpus on a standard workstation motherboard can also be a challenge due to a severe shortage of space between them. Dual slot spacing between PCIE x16 slots is a tight fit as the tubes can take up almost three slots worth of space between cards. In this configuration they are installed on alternate slots so if you had 7 slots in total you’d only install on 1, 3, 5 and 7 leaving 2, 4 and 6 empty. Though 2, 4 and 6 will usually be PCIE x8 slots anyway as opposed to the rest being PCIE x16.

As you can see below these cards have been completely stripped down of their air cooling and heatsink apparatus prior to attaching waterblocks and tubing for coolant to pass through them. The tubing is secured using barb fittings which are stronger than the alternative: compression fittings though they do lack the aesthetic appeal of compression fittings. Compression fittings can come apart under tension and that can create a real mess as I realised the hard way one night.

If not using full card waterblocks (which these aren’t) individual adhesive heatsinks for all the ram chips (known as ram sinks) are required for sufficient cooling. There may be upto 12 of these tiny little ram sinks on each face of a card. I don’t have any photos of that right now but I’ll try and get some. Though, ram sinks, can be flimsy and easily become dislodged and fall off the cards if they are knocked which is why some people prefer full cover blocks. Full cover blocks are more robust but also more expensive.

In terms of power the system is supplied with 2400 watts of power composed of two 1200W power supplies chained by an adapter for the first to kickstart the other on boot. Half the gpus are powered by one and half by the other. This particular machine has 128GB of RAM and dual xeons with a combined total of 32 cores. Update: As rightly pointed out by the commenter below I meant hardware threads not physical cores here.

Note: I do not own the hardware or the photographs but consent has been acquired to publish them here. Also this system is not conceived or assembled by me though I do have the pleasure of loading it with OpenCL benchmarks and computations.

11 thoughts on “8 GPU watercooled computation rig using Power Color Devil 13 HD 7990 cards

  1. Full-cover blocks were unfortuantely not available for the PowerColor 7990 (nor are they likely to be given the small number of cards sold) so individual blocks were the only option. Some modification was required to cut the three slot brackets down to two slots and shortening the mounting tabs on the waterblocks to physically fit on the cards. Reverse side ramsinks would not have been required if the original backplates could have been retained, but this was not possible due to the very tight clearance between the CPU0 RAM slots and the first PCIE slot on this motherboard (common issue with dual CPU EATX boards). Note that this machine only has 16 cores – the most cores you can get in a single Intel socket is 10 and even that’s only available on the extremely expensive 4-socket machines – but it can run 32 threads via hyperthreading. AMD do make a 16-core processor but for nearly all workloads it’s considerably slower than the top end Intel 8-core.

  2. In terms of power, cooling and performance it works perfectly. However the motherboard has an issue with running out of pci resources with a certain number of gpus or above. We are working with Asus to get this resolved.

  3. hey man whats the name of the water blocks you used for the powercolor devil 13 7990?? please let me know thank you brother

  4. What brand/model are those water blocks? I’m looking to set up dual Devil 13s with liquid cooling. This would be amazing info! Thanks so much in advance!

  5. I didn’t build this – my ex-colleague did and it was quite some time back. So I can’t quite remember. But do a search and you should find some. I’m pretty sure EK said some time back that they were working on some waterblocks for the 7990.

  6. EK has waterblocks for the PowerColor Devil13!

    Four of those things with a 4-way EK Bridge would be SICK. EK also sells two slot brackets for these things.

    By the way, I would do the setup like this:

    Water temperature is pretty much constant throughout the loop, providing that you have suffucient pump power. With two D5’s, you should be fine. A setup like this just cleans the system up and removes excess tubing.
    The Quad EK-FC bridges are semi-parallel, so you’ll have the same idea as now and not 8 GPU’s all behind eachother.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s