local ai / plex server | Self-Hosted | Page 1

outer flower Apr 23, 2024, 6:55 PM

#

16 drives with no HBA
2 m.2
2 3090
1 a380 single slot mod

#

Will it overheat that’s the question

#

nimble jasper Apr 23, 2024, 6:59 PM

#

How do you have 16 drives connected with a HBA?

#

Your 3090s look ripe for overheating, especially the upper one

outer flower Apr 23, 2024, 6:59 PM

#

Slimsas to 8 sata

nimble jasper Apr 23, 2024, 7:00 PM

#

Oh the slimsas is built into the motherboard?

outer flower Apr 23, 2024, 7:00 PM

#

Yeah

nimble jasper Apr 23, 2024, 7:00 PM

#

Is this the Define 7 XL?

outer flower Apr 23, 2024, 7:00 PM

#

Meshy XL but same thing

#

I don’t know what to do about the heat.. I have 3 intake fans on the front

#

1350w psu

nimble jasper Apr 23, 2024, 7:00 PM

#

The only viable option to fix overheating is watercooling

outer flower Apr 23, 2024, 7:01 PM

#

To be honest they will be at idle unless doing chat or stable diffusion

#

And it’s all hobby so nothing insane

nimble jasper Apr 23, 2024, 7:01 PM

#

I have 10 spinning rust drives and a 32 core threadripper that I have watercooled in my Define 7 XL. I'll be adding a watercooled RX 5700 XT soon

outer flower Apr 23, 2024, 7:02 PM

#

Nice!

nimble jasper Apr 23, 2024, 7:02 PM

#

Well sure, but your idle temps aren't the concern

#

The load temps are

outer flower Apr 23, 2024, 7:02 PM

#

I know. We shall see how bad it gets. I feel like I bit off more than I could handle with this build

#

Got this bad boy for backup power

#

nimble jasper Apr 23, 2024, 7:08 PM

#

If you do end up watercooling - I think the 3090 PCB itself is short enough that it doesn't go past the end of the motherboard. It's just the air cooler that extends past that
I was able to mount a Aquacomputer Ultitube reservoir + D5 pump in between the motherboard and the hard drive stack, with the screws going through the Fractal cutouts in the metal panel separating the main chamber from the PSU basement

#

I don't know how the PCIe slots are laid out on that motherboard, but you could get some breathing room for your top 3090 if you move it up one slot and then use a PCIe extension to put your A380 in the vertical GPU mount area

#

You then also wouldn't need your A380 to be single slot

outer flower Apr 23, 2024, 7:14 PM

#

nimble jasper Apr 23, 2024, 7:15 PM

#

Yeah so you could have a 3090 in slot 1, PCI extension in slot 4 to the vertical mount A380, and 3090 in slot 5

#

Dunno if you have full x16 lanes in slot 1 or even if you need the full lanes for your use case

outer flower Apr 23, 2024, 7:17 PM

#

they are all full 16x or 8x

#

I think I do to make the llm's faster. The a380 takes a 16x slot

nimble jasper Apr 23, 2024, 7:18 PM

#

Wait what

#

You need x16 for the A380?

#

I thought that was just for display output and the LLMs run on the 3090s?

outer flower Apr 23, 2024, 7:19 PM

#

I could probably run it on a 8x to 16x pcie riser cable

#

but i'll be using the a380 for plex transcoding

nimble jasper Apr 23, 2024, 7:20 PM

#

Oh pfft you don't need anywhere near even PCIe 4.0 x8 for Plex transcoding

outer flower Apr 23, 2024, 7:20 PM

#

yeah i just meant thats how big of a slot it needs

#

but then i remembered they make 8x to 16x slot size wise

nimble jasper Apr 23, 2024, 7:20 PM

#

Huh

#

You can put a x16 device in a x1 slot and it will work

#

At reduced bandwidth obviously

outer flower Apr 23, 2024, 7:21 PM

#

right

#

nimble jasper Apr 23, 2024, 7:22 PM

#

Ok, so why are you concerned about getting a x16 slot for your A380

outer flower Apr 23, 2024, 7:22 PM

#

i've never built anything other than a gaming computer, learned a lot doing this

#

I guess im not, I just used the slots and cables that I had

#

I could have used the middle 8x and vertical mounted the a380

#

i even have the cable coming in the mail

nimble jasper Apr 23, 2024, 7:22 PM

#

Yeah I'd do the vertical mount via extension to get some breathing room. Far less drastic than watercooling lol

outer flower Apr 23, 2024, 7:23 PM

#

right lol

#

im gonna boot her up this way, see how bad it is, and if its bad then i'll try that

#

this just looks cleaner

nimble jasper Apr 23, 2024, 7:23 PM

#

Can't argue with the cleaner look

#

IIRC the founders 3090s use the through card cooler so your bottom 3090 will be feeding hot exhaust directly into your top 3090

outer flower Apr 23, 2024, 8:02 PM

#

you would think they would have planned that better, they built them to use nvlink

nimble jasper Apr 23, 2024, 8:07 PM

#

I don't think I've ever heard of anyone using that though

#

For gaming it's essentially dead

#

For non gaming, applications generally seem to manage splitting the workload across multiple GPUs on their own

#

And besides, they probably figured if you have the money for more than one 3090 you can afford to watercool it anyways

outer flower Apr 23, 2024, 8:47 PM

#

70c on the top gpu 50c on the bottom

#

I haven’t offloaded plex yet

#

Unraid kernel still doesn’t support arc cards

#

Waiting for 6.13

outer flower Apr 23, 2024, 9:16 PM

#

nimble jasper And besides, they probably figured if you have the money for more than one 3090 ...

How big are the radiators? i've got room at the top of the case

nimble jasper Apr 23, 2024, 9:17 PM

#

What kernel is unraid using?

#

Internally I have a 60mm thick 360mm rad, but I am also using an external 420mm x 420mm rad https://shop.watercool.de/MO-RA3-420-PRO-stainless-steel_1

Watercool MO-RA3 420 Pro Stainless Steel, 289,95 €

Our MO-RA3 monster radiator is one of the most flexible and powerful radiators in the world. TOP workmanship combined with top materials make the MO-RA3

outer flower Apr 23, 2024, 9:19 PM

#

6.1

nimble jasper Apr 23, 2024, 9:19 PM

#

Doubt that a single 360mm or 420mm is going to work for two 3090s

outer flower Apr 23, 2024, 9:19 PM

#

Guess i could just liquid cool the top one

nimble jasper Apr 23, 2024, 9:19 PM

#

Oof 6.1. I'm surprised they're that behind when LTS is 6.6

#

Yeah. Though honestly 70C is fine

#

I think they throttle at 80C

outer flower Apr 23, 2024, 9:20 PM

#

its idling down to 50c running codeproject ai and plex

#

bottom idles at 40c

nimble jasper Apr 23, 2024, 9:20 PM

#

Was this under your LLM load or a stress test like Furmark

outer flower Apr 23, 2024, 9:20 PM

#

probably not worth the effort

#

just regular llm load

#

no benchtest

nimble jasper Apr 23, 2024, 9:20 PM

#

Ahhh that's why

#

I'd probably just do the A380 vertical mount and call it a day

outer flower Apr 23, 2024, 9:21 PM

#

that 1 slot make that big of a difference you think?

nimble jasper Apr 23, 2024, 9:21 PM

#

It's choking for air and being fed hot air

#

I've seen the temps on old dual/triple/quad SLI configurations on air cooling

#

The middle cards always had much higher temps

outer flower Apr 23, 2024, 9:22 PM

#

ight well i'll get one ordered then for the a380

#

8x to 16x

#

thanks for the sugestions!

nimble jasper Apr 23, 2024, 9:23 PM

#

It doesn't have to be 8x to 16x btw

#

Can be 8x to 8x

outer flower Apr 23, 2024, 9:23 PM

#

just has to be long enough slot wise yeah?

#

is 8x bandwidth or size or both?

#

i've never messed with pcie this much

#

just always slapped stuff in

nimble jasper Apr 23, 2024, 9:24 PM

#

No it doesn't even have to be long enough

#

You could use a 16x card on a 1x to 1x extension and it would still work, just bandwidth limited

#

You would need the extension cable to have an "open back" though

outer flower Apr 23, 2024, 9:24 PM

#

interesting, they are desnged to just hang off the side?

nimble jasper Apr 23, 2024, 9:24 PM

#

Your motherboard's slots are not open back

outer flower Apr 23, 2024, 9:26 PM

#

#

that wont work right?

nimble jasper Apr 23, 2024, 9:27 PM

#

What I mean by open back https://www.amazon.com/XRIKUI-90Degree-Extension-Express-Extender/dp/B0CMHGHD49

XRIKUI Dual 90Degree Right Angle PCIe 3.0 x1 Extension Cable High S...

XRIKUI Dual 90Degree Right Angle PCIe 3.0 x1 Extension Cable High Speed PCI Express Gen 3 1x to x1 Riser Extender 10cm

outer flower Apr 23, 2024, 9:27 PM

#

oh i see

#

slot is open

nimble jasper Apr 23, 2024, 9:27 PM

#

Idk what the technical term is for that but that's what I'm talking about. The card can just hang off the end

outer flower Apr 23, 2024, 9:27 PM

#

gotcha

nimble jasper Apr 23, 2024, 9:28 PM

#

I've "converted' slots to open back before by taking a Dremel with a steady hand

outer flower Apr 23, 2024, 9:28 PM

#

lol!

nimble jasper Apr 23, 2024, 9:28 PM

#

But if you aren't careful you can damage the pins

#

Probably better with a hot knife or something

outer flower Apr 23, 2024, 9:28 PM

#

i'll just get one long enough

nimble jasper Apr 23, 2024, 9:29 PM

#

Yeah for a riser it's just easier to get either the right length or an open back

#

Sucks when you need the slot on the motherboard itself to be open but it's not

#

I honestly don't understand why manufacturers do that

outer flower Apr 23, 2024, 9:52 PM

#

probably so that people dont slam a gigantic card into a 1x slot and then complain when the slot breaks off or something

#

https://www.ebay.com/itm/304855558887?itmmeta=01HW6FEW8RZC7SAN5HB3WJ4F33&hash=item46faceaae7:g:IM8AAOSw3ndkHYAR&itmprp=enc%3AAQAJAAABAKQjYzWM8tK705IjkjJnpaH%2F01tuoNgf86BddYlAFn4my9xzyjM8HJonFzBXW%2BmfC6%2BBDtaLe0q4V45bB9PAhRoUcf%2FnJeutEYhpUNd1INwuT31wMbJNSkrWPY0NlyBOuz2M%2FPyehoH1%2FnUij%2FwLDlWb5wJYEUFABvHKmBOZfgWqSdFC6rQINbJrF3CtAxRbqKHaTmnJGGmX4cbuoHDllGHR%2FFHMT4UdLnD3I01k%2B8l2Uoc990oiyNOtNt3LAWAJmho9f1Q7bA6nZP8R5fG%2F77IoQBxZ7%2FqGFxeJoei9L0K%2FIDLgN9tvVwxWHnpv7lM%2BY3onXRL%2B%2BxZNkOKmIu33dw0%3D|tkp%3ABFBMyMS7z-Fj

eBay

PCIE 4.0 X8 to X16 Extension Cable Plated GEN4 Lossless PCIe 8x Rig...

PCIE 4.0 X8 to X16 High-speed Extension Cable Silver Plated Cable GEN4 Lossless PCIe 8x Right Angle 90 Degrees. LOTES GEN4 connector, gold plated 15U. Supports PCIE 4.0 devices and is backward compatible with PCIE 3.0.

#

290w idle overall

outer flower Apr 23, 2024, 11:01 PM

#

@radiant island that slim sas we have on our mobo's is sick. 8 sata drives

radiant island Apr 23, 2024, 11:01 PM

#

Yuuuuup

outer flower Apr 23, 2024, 11:02 PM

#

can do 16 sata drives, who needs an HBA?

radiant island Apr 23, 2024, 11:14 PM

#

You sure that slimsas can do 8? I could only find a cable for 4

outer flower Apr 24, 2024, 12:11 AM

#

I’m using all 8

#

Too 8 drives so 7 right now

#

outer flower Apr 24, 2024, 12:13 AM

#

radiant island You sure that slimsas can do 8? I could only find a cable for 4

DiliVing SlimSAS X8 to... https://www.amazon.com/dp/B09S3NKZNZ?ref=ppx_pop_mob_ap_share

DiliVing SlimSAS X8 to 8*SATA,SFF-8654 74P to 8*SATA 7P, SlimSAS Ho...

This SAS cable can be used to connect internal devices, e.g. a hard disk with SATA 7pin connector to a controller with Slim SAS SFF-8654 8X connector. Ultraport SlimSAS 8X 74Pos Cable Assembly. SlimSAS 8X Straight to SATA 7pin. Length: 80cm. Connector 1: Slim SAS 8X 74pin. Connector 2: SATA 7pin....

radiant island Apr 24, 2024, 12:13 AM

#

Oh sick

outer flower Apr 24, 2024, 12:13 AM

#

Yeah it’s actually amazing

#

Built in HBA lol. Too bad the -i doesn’t have the sas controller. But I’m all sata anyways

radiant island Apr 24, 2024, 12:14 AM

#

#

Ha past Alex was wise

outer flower Apr 24, 2024, 12:15 AM

#

lol!

#

Now you gotta find it

radiant island Apr 24, 2024, 12:15 AM

#

Oh it’s in the box for sure

outer flower Apr 24, 2024, 12:15 AM

#

I was always worried the fan on my HBA would fail

radiant island Apr 24, 2024, 12:15 AM

#

I wouldn’t have used something else. Just forgot the details

outer flower Apr 24, 2024, 12:15 AM

#

Sliger case has 10 drives?

#

Almost covers the case

radiant island Apr 24, 2024, 12:16 AM

#

Yes

amber comet Apr 27, 2024, 7:48 AM

#

outer flower I don’t know what to do about the heat.. I have 3 intake fans on the front

Maybe store it lying down instead of standing up?:)

outer flower Apr 28, 2024, 3:45 AM

#

amber comet Maybe store it lying down instead of standing up?:)

Would that help?

amber comet Apr 28, 2024, 7:00 AM

#

Heat wouldn’t dissipate into the top graphics card if you store it horizontally.

nimble jasper Apr 28, 2024, 5:09 PM

#

?

#

The graphics cards have fans pushing air out and sucking air in

#

Those fans make the natural rising of hot air insigificant

outer flower May 9, 2024, 6:58 PM

#

@nimble jasper took your advice. Dremel out the 8x and made the move.

nimble jasper May 10, 2024, 5:06 AM

#

How are the temps now?

outer flower May 15, 2024, 6:09 AM

#

50-60c idle for both gpu's

#local ai / plex server