36TB Dual E5-2670 Build


Recommended Posts

Hello Everyone!

 

I just recently joined the community, and with help from @Lev (very, very helpful!) I was able to get some decent hardware for my server (Hardware decision thread). Here are the specs:

 

  • Supermicro 4U Server
    • CSE-846A-R1200B Chassis

    • X9DRI-F Motherboard

    • 2x E5-2670 2.6ghz 8-Core 8.0 GT/s / 20mb Smart Cache CPUs

    • 8x 8gb PC3-10600R Server Memory

    • 24x 3.5" Trays

    • SAS2-846EL1 Backplane

    • LSI 9266-8i 1mb Cache with Battery Back (I got mine working with unRAID)

    • LSI 9207-8i

    • 2x 1200w PSU

 

I had a bunch of drives laying around and this is what I am currently running:

  • 13 Total Drives
    • Array
      • 2 Parity (4TB)
      • 9 Data (1.5-4TB)
        • Total usable space: 28TB
    • Cache
      • 500GB SSD (VMs, docker, cache for array, etc.)
    • Unassigned
      • 500GB (For live TV recordings and downloads)

 

I just wanted to give the community some general thoughts on my process, as well as some of the hurdles that I came across. All feedback is welcome :-). In no particular order:

 

  • The case, it's BIG. Like really big. (Also its heavy, without HDDs its ~70 lbs.) I have never worked on server hardware before, and this was an experience. 
  • Before I installed anything, I admired this wonderful piece of hardware. Chassis open detection is kinda cool. There was also a giant plastic shroud over the top half of the motherboard. A quick google search said to leave it, it helps with airflow over the CPUs. 
  • No spare molex or any power inside. If I get a GPU that needs 6 pin or 8 pin I'm going to have to do some work-arounds.
  • IPMI. It is amazing! I will never work without it again! For anyone who is debating if this feature is worth it, it is 100%. 
  • I decide to fire this bad boy up. Wait, where is the power button, WTF? Google revealed that its on the left on the handle-grip thingie. My server came with it, but not installed. I had to play with the cable for a bit but eventually I got it working.
  • It's loud. Like loud. Yes you can play with fan speeds (and it can get louder) but still loud. However, I have the server tucked away in the basement so noise isn't an issue.
  • Dual PSUs. Interesting. I took one out though I don't think I will need it
  • Backplanes are interesting. Definitely easier to work with than individual cables. I need to some more research, but I wonder how to I hook up more hard drives that the LSI-9266 doesn't handle?
  • The LSI 9266-8i. This was interesting to say the least.....
    • Short version: It works.
    • TL;DR
      • It is based on the SAS2208 chipset. Some people say it won't work with unRAID. Some say it will. Some say it will but you will have problems. Played with the bios and the CLI, but it would not do JBOD. Determined, I scoured the forums and google. In the end, I updated to the latest firmware, than ran a simple command from the CLI to make it work in JBOD (note, after the command I can't access the card's bios, but who cares?). I got the info right from the manufactures website, as it is a supported feature. However, after running a non-correcting parity check, I did get 5 errors when before I did not. Need to explore more.
      • EDIT: Now using a 9207-8i. See posts below.
  • I had unRAID running on some old AMD hardware the past week just to learn and play with it. Made a bunch of dockers and a few VMs. When migrating over, unRAID booted up nicely and gave me no trouble. The VMs however...
  • VMs. I had 3 built: I did not have them set to auto-start because I knew i would have to tweak cores and pass thru.
    • CentOS - Freaked the hell out. Kernel panics, nothing working, everything went kaput. Had to rebuild. Was able to retain my settings so in the end no big deal.
    • LibreELEC (not the LimeTechnology image. Followed @gridrunner's video (link) )  - Reassigned passthru and no problems. I have a GT610 that worked flawlessly. I was even able to do HEVC, which slightly surprised me becuase my AMD hardware would not. Then again its old hardware so maybe it shouldn't surprise me.
    • Windows10 - Still freaking out, will most likely rebuild because I had nothing too important in there
  • IPMI. Yes this deserves a 2nd bullet, I love the feature that much!
  • Docker. Marvelous! 

 

That's all I can think of for now. If I can think of anything else, I'll update this post accordingly.

 

EDIT: After some trial and error, I have discovered that if I turn on my LibreELEC VM (w/ GPU passthru) and my TV is not on the correct input, the entire machine crashes and requires a hard reboot). Searching the forums and google, I was advised to turn on 4G (or something) and update bios. Will play with and report back

 

EDIT2: Turned on 4G decoding in bios and the crash/error went away. Which is good because I am not a big fan of updating the BIOS unless there is a really, really good reason for it.

Edited by CrimsonTyphoon
Link to comment

Love the post! And welcome to the community. I recently stopped lurking after years and finally found the confidence that I knew just enough information that I could contribute.

 

I love your post cause it's also a mirror to my experience as well over the last month upgrading to a used enterprise level system. This is my third UnRaid server build, since ... wow .. just looked in my emails to check the date. I bought my first Pro Key in 2011! And I totally agree with your comments, I wish I would have upgraded to this level of equipment much sooner. <3 IPMI

 

A few things I've learned in the last month, and a few opinions, use or ignore:

 

  • "No spare molex or any power inside" yeah it's really clean inside. I did have spares, they were hidden underneath my motherboard slide out tray. But I have the 36 bay chassis so it's a bit different I imagine than yours. I'm sure theirs a official SMC part to for it. I found that anything I can think of, they already have and their online store is super fast in shipping me anything I need. SMC tech support over email has also been great. <3 Sonny @ SMC
  • "It's loud" yeah, it is. I'm not going to bother swaping out the fans for something else. I'd rather try the fan speed method method through IPMI command line as others have posted here in the forums. Let me us know what you find that works for you.
  • "Dual PSUs" I agree, I don't really fear one dying so kind of over-kill for home use. But I really like it for the so reason that I can keep the server powered up, and move the power cords, one at a time from one plug strip or power source to another. Very handy! Also I hooked upa  Kill-A-Watt Meter and noticed that it splits the power load across both PSUs. So if you have two power circuits as I happen to do near your server, you can take advantage of that.
  • "Backplanes are interesting." Yes! We have the same backplane. My HBA was only single linked to mine, I had to lookup how to add the extra cable to get dual link. This improved my parity checks from 115MB/sec to 180MB/sec which is the limits of my hard disks.
  • "LSI 9266-8i" I'm very happy you got it to work, I saw the same posts and was discouraged. Simply too many other great choices that are 100% proven. Don't hesitate to swap it out if you have doubts. I've had many m1015 (LSI SAS 2008) cards perform solid, and I quite like my new LSI 2308 adapter: It's PCIe 3.0 dual linked to the same backplane you have for supporting up to 24 drives.
  • "VMs" as you experience they are just a PITA. I was happy when I finally replaced my last one with the final Docker container I needed thanks to @bonienl fantastic work adding improved networking features. I've turned off the VM service entirely.

A few suggestions if you don't already have, or for when you've reloaded your budget!

  1. Get a rack mount UPS. Power outages are painful. I spent quite a bit on a very large UPS last year and it's saved me countless hours of trouble. Cable Modem, Router, Switch, PoE Cameras, PoE WiFi Access Points, UnRaid, everything stays online now for 6+ hours saving all the time of having to reset and booth each device in sequence one at a time after an outage.
  2. Get a rack! I bought a simple 4 post 42U rack from StarTech, Love it!
  3. If you get a rack, get a PDU that's fully switchable. Being able to turn on/off power to anything plugged into it is the next best thing after IPMI, or when IPMI fails and needs a hard reboot!

Welcome again to the community! Great to have you!

 

 

Link to comment
On 8/18/2017 at 0:48 AM, Lev said:
  • "Backplanes are interesting." Yes! We have the same backplane. My HBA was only single linked to mine, I had to lookup how to add the extra cable to get dual link. This improved my parity checks from 115MB/sec to 180MB/sec which is the limits of my hard disks.
  • "LSI 9266-8i" I'm very happy you got it to work, I saw the same posts and was discouraged. Simply too many other great choices that are 100% proven. Don't hesitate to swap it out if you have doubts. I've had many m1015 (LSI SAS 2008) cards perform solid, and I quite like my new LSI 2308 adapter: It's PCIe 3.0 dual linked to the same backplane you have for supporting up to 24 drives.

 

I have had my parity drive red ball in the past 24 hours (thread). My gut is telling me that its the LSI-9266 controller because it has been giving me random problems here and there over the past few days. Rather than pull my hair out, I'll simply get a new card. I presume I should go wit ha LSI 2308 card that can fit 24 drives for my expander? Also, how do I make sure it is in dual link mode? Actually, to make things easier, if you could tell me the exact card that you have @Lev, I'll go and buy it :-)

 

Besides this, the server has been GREAT. Got my hang on VMs, just waiting for my 1050Ti to show up :-)

15 minutes ago, gridrunner said:

@CrimsonTyphoon  interesting post. And welcome to the community now you  are officially an unRAIDER !

 

Glad to be part of it! I have a few more random thoughts/ideas over the past few days, will update this post when I get time to put my thoughts into words :-)

Edited by CrimsonTyphoon
Link to comment

Since I know you use ebay, search on 9207-8i and make sure it comes with the full-height bracket or both brackets. Some tips for when you receive the card and these are good for any of the typical IT-IR LSI cards.

  • Visually inspect it closely as soon as you open the package, look for damage or scorch marks.
  • Take picture of the front and back of the card. As you know already, you'll need that sa address number later when you flash it
  • Blow it clean with canned air, very cheap at costco or amazon
  • Install and your server, but don't attach any cables, just see if it posts as you'd expect.
  • reboot and then when it posts again, hit he ctrl-c and go into the card's setup. Go to the details page where it lists the sas address and firmware versions. Take a picture of this screen. Confirm the sas address on the screen matches the picture you took of the card.
  • Even if it says its running the most current firmware, flash it anyway. Follow the flash steps from the other thread you followed before.
  • Once you've flashed successfully. Again go into card's setup and check firmware versions and sas address is as you'd expect.
  • Power down and attach a sas cables to the card, everything should work smooth throughout the UnRaid boot up. Login thru the browser and check the logs

See this thread for details on dual-link with that backplane, it helped me a lot to get mine working as my server didn't ship with it already setup, yours might have. Run the command in that thread, you may not even have to open your case and look if it says you're already in dual-link.

 

 

  • Upvote 1
Link to comment
11 hours ago, Lev said:

Since I know you use ebay, search on 9207-8i and make sure it comes with the full-height bracket or both brackets. Some tips for when you receive the card and these are good for any of the typical IT-IR LSI cards.

 

I really don't know what I would do without you! Purchased :-)

 

Once the card comes I'll look into verifying that I am in dual link mode.

 

Thank you :-)

Link to comment

Just wanted to post a small update:

  • Even thought I was able to get the 9266-8i working, it wasn't worth the trouble. On a parity sync, I had errors. I had some errors in my VM's and it told me my parity 1 drive was bad (its not). So i took @Lev's great advice and went with the 9207-8i
  • The 9207-8i came a few days ago. Wasn't able to install it right away but finally got the chance:
    • Took out the 9266-8i, plopped in the 9207-8i w/o drives attached. Verified the S/N matched (it did). Card came with the P20-IT firmware so I did not reflash it (If it ain't broke, don't fix it!)
    • Plugged the 9266-8i back in [and took out the 9207-8i]. Went back into the MegaCLI and put the card back into RAID mode. Verified it worked no problem.
    • Swapped cards again (final time). Hooked up my drives and it worked without problem. I then accidentally disabled the card, laughed at my mistake and re-enabled the card.
    • unRAID working perfectly. Will rebuild parity 1 overnight
  • I might have mentioned this before but... I LOVE IPMI!
  • Ran the command in this thread in regards to dual link. Happy to say that I got the correct result. @Lev to the rescue again!
  • Took some power readings on my Kill-A-Watt (specs in post 1 of this thread):
    • Baseline:
      • CPU Scaling Governor: Power Save (unless noted)
      • Fan Speeds: Normal (50%)
      • Three (3) VMs running out of 4 all off cache SSD
      • Twenty (20) of 22 Dockers running all off cache SSD, with 3 accessing unassigned disk.
    • All drives powered down: ~157 W
    • All drives powered up: ~198 W
    • All drives powered up, CPU Scaling Governor: Performance ~222 W
    • Parity rebuild (not check), CPU Scaling Governor: Performance ~240 W
    • I will update power readings after I do some more testing (i,e. Parity check, Parity Rebuild, maxing VMs, etc.)

 

On 8/21/2017 at 10:19 AM, kizer said:

No pictures no proof. :D

 

Post some up when you get a chance. We like pics. 

 

Will upload pictures of this beast within the next day or two :-)

Edited by CrimsonTyphoon
Add wattage for rebuild
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.