Please read my previous post on my DIY NAS here: https://www.devroom.io/2020/02/28/building-a-diy-home-server-with-freenas
A brief history#
In May 2019 I decided it was time to jump on the NAS bandwagon. I started out with the following:
- Fractal Design Node 804
- 4x 3TB WD Red (retail)
- Gigabyte Z370M D3H
- Intel i3 i8350K 4c/4t CPU
- 16GB Non-ECC Memory
- 128 GB Gigabyte SSD Boot drive
- FreeNAS with a single RaidZ1 (4x 3TB) pool
That’s when it all started. Then things got wild.
Update November 2020#
So, how are things today.
- Norco 4U, 16x 3.5" LFF Hot swap case
- 4x 3TB WD Red (retail)
- 4x 8TB WD White (shucked)
- 4x 14TB WD White (shucked)
- 1x 3TB WD Green (retail)
- 2x 120GB Crucial BX500 SSD’s (because why not?)
- SuperMicro X10SRi-F motherboard
- Intel Xeon E5 2620v3 6c/12t CPU
- 64GB (2x32) ECC Memory
- 2x Dell PERC H200 HBA
- 250 GB Samsung SSD Boot drive
- Proxmox 6
So, aside from the original 4x 3TB WD Reds, everything has changed. From enclosure to operating system.
Are you done now?! What more could you want?!#
Nope. Never. I have some parts on the way to setup 10Gb ethernet between my NAS and Ryzentosh work station. This is one of the reasons I upgrade the motherboard and CPU - to accommodate 2x PCIe x8 HBA’s and 1x PCIe x8 10GB Ethernet.
Norco 4U Rack with hot swap bays#
Great choice. It’s much easier to swap drives if need be. But I also have 4 empty bays right now. This is great to test a new disk or copy stuff over from old cold-storage disks. It also helped a lot migrating to the new SuperMicro motherboard.
Because the case is bulky, it also offers lots of room for 120mm fans. And 120mm fans (especially Noctua ones) are quiet.
Motherboard + CPU + Memory#
My 4 core / 4 threads i3 8350K has served me well. It’s quite powerful too. But when running about 20 VM’s / Containers I feel it’s getting spread too thin. That, and the lack of PCIe slots on the micro-ATX board were a reason to look for something new.
There are two options if you want lots of PCIe slots: high-end gaming motherboards that support 3 or 4 GPUs or (used) server hardware. Since this is turning into a proper home-lab server, I decided to go with used server gear. It’s a bit cheaper and more suited to the task.
After scouring eBay and local classified sites I found a really good deal on a SuperMicro X10SRi-F. It has quite a lot to offer that makes me very happy:
- IPMI remote management
- 10x SATA3 on board
- Lots of PCIe slots:
- 1x PCIe 3.0 x16
- 2x PCIe 3.0 x8
- 1x PCIe 3.0 x4
- 1x PCIe 2.0 x4
- 1x PCIe 2.0 x2
- Dual gigabit ethernet
The accompanying Intel Xeon E5 2620v3 6c/12t CPU has a lower clock speed of 2.4GHz (boost to 3.2GHz) as opposed to the 4.0GHz of my previous i3 8350K, but it does offer 6 cores instead of 4 and supports HyperThreading. So that offers a lot more flexibility with running a multitude of VMs and containers.
I also added 2 stick of 32GB ECC Registered memory for a total of 64GB. It’s enough for now, but with LRDIMMs, this motherboard has 8 DIMM slots available for a maximum of 512GB of memory. Quite enough for now and room to expand. Great!
Proxmox#
Proxmox is great if you want to have a ZFS pool and run lots of services on top. As discussed in my previoust post, FreeNAS is great, but it’s not made for running additional services. Jails restrict you to FreeBSD and the bhyve hypervisor is not quite mature yet to easily run VMs.
Proxmox is just Debian linux with a lot of goodies for running a homelab server. ZFS on Linux being one of them. First class support for containers and virtual machines is another.
The only thing ’lacking’ from Proxmox, coming from FreeNAS, is easily setting up file sharing. But in the end all I needed was to spin-up a container, install Samba and configure the shares I wanted to create. Voila. Nice shares and TimeMachine backups.
10GB Ethernet#
Yeah. That’s happening. I find myself storing more and more data on a network share on the NAS. But with lots of audio and video files, 1000mbit can be limiting (e.g. 120-ish MB/s).
I found myself a pair of Solarflare SFN5122F 10G Ethernet Dual Port (SFP+) cards on eBay. These should even work under macos, which is what I’d need. They’re still in the mail right now - but it should be pretty straightforward setting these up for a direct 10G link between my Ryzentosh and NAS.