Does Ceph really have mad HW requirements?
I wasn't sure whether to post this in here or r/ceph but as this setup will be a homelab, let it be here.
I'm planning to build a 4-5 node Proxmox HA setup with second hand rack servers that I happen to have. Enough CPU and ECC RAM for everything. Not yet 10 gbit/s but it's easy and cheap to fix. I will use hardware raid-1 with old SAS spinners for the boot and the system.
Now my problem is that I've become convinced that I should learn Ceph and it would make a great storage platform for VMs, and I don't want to do it hyperconverged but separate. So many complaints about performance if hyperconverged, which I completely understand and these servers don't have much spare power after my vision of VMs are running. Let's say I need ~5 TB of fast usable space, so NVMe it will be. I also want the whole setup to eat less than 1.5 kW of power under average load, while the rack servers take like 200-250 W each. At first I thought this would be easy. You can get cheap and low power mini PCs with SFP+ (for 10Gbe) from China and ~4 TB m.2 pcie NVMes aren't too expensive for my budget. Another five nodes of Ceph built with these would do all I want nicely. But then:
1) No ECC support on these machines. Alright. Maybe I can live without. But wouldn't want to.
2) Everywhere I read people are telling that with Ceph you need a datacenter NVMe with PLP. Not cheap anymore but alright, maybe I could afford that.
3) Now the dimensions of a datacenter NVMe with PLP: always 22100 and all these tiny machines take only 2280, not physically possible to make any longer to fit in.
There is Addlink D60 1920GB which is 2280 and costs $200+ (which is high $/Gb for me) and it's the biggest they have - no ~4 tb available.
Am I right in my conclusion that the only way to accomplish this with Ceph is to use a server grade motherboard, cpu and ram (for the ECC) for nodes and then add pcie to m.2 adapters supporting the 22100 physical format? And then give it another 5x 150-200 W when all I want is to use 5 little (just not the most tiniest) NVMe sticks? Is there something wrong if I feel that 750-1000 W for just this is insane?
Are there any less power hungry and cheaper options, keeping the IOPS and data integrity as high as possible?