It is really fun. I like having enough stuff around to tinker. It's easy to forget how hard it is to do anything without so much entrenchment and tools, then go to an ordinary house and try to solve some computer problem, and it feels like living on a desert island. Computers don't work that well in general.
However, it's noisy. That may kill the whole thing soon. I don't know.
Future things we ought to work on:
It has lots of fiber gigabit ports, because in the used market these are cheaper to get than copper ports (the NIC's are cheaper, too).
the main reason for buying it was to support some iSCSI stuff I am doing with terabithia and fishsticks. for the next generation music and movie storage thing, /arrchive.
ezln, kadmirra, antioch, castrovalva, fishsticks
The fishsticks are the grey things on the extreme right. They are full of about seven hard disks each. At least one hard disk in each fishstick is an iSCSI target used by terabithia for mirrored ZFS. The fishsticks netboot from terabithia using a custom 'genkernel' script, and Etherboot burned into a FLASH ROM that I bought on eBay and added to the Compaq NC6136 (i82543) gigabit NIC's on which I've standardized here. (there is no Intel PXE image for these NIC's, and they don't ship with FLASH.)
We built them by getting a cheap tower case marketed for loading up with seven CD-ROM's (SCSI lets you have seven, remember?). Then we put seven PATA disks in the old fishstick, seven SATA disks in the new fishstick. I'm surprised they still make cases for this niche, but it turns out to be the perfect cheap way to mount disks, in my opinion, and give them proper ventillation. Each disk goes into a 3.5"-to-5.25" bracket and has a vented metal faceplate in front of it. In one of them we dremmeled out the top of the case and added a filtered, high-flow (~$10) fan from Allied. We never got around to adding a fan to the other one, so we leave it without filters, and it gets a bit dustier. They're running Gentoo Linux, which I hate, but BSD doesn't have any filesystems I feel are appropriate for large volume sizes. Each disk has a separate filesystem on it, no RAID. This no-RAID is a carefully-thought-out decision which I will defend pretty vigorously. It's not an unconditional preference, but in short we can't afford to spend what it takes to do it safely.
In the pizzabox tower there's castrovalva, another shell server and a hackint.org irc server. castrovalva is a Digital DS10 running an old version of NetBSD with horrible pthread bugs. There's antioch, another shell server, running I think NetBSD on a SPARCstation 10. ``shell'' means web, mail, DNS, 'screen' sessions, a working C compiler, things like that from the old Internet.
Above them unplugged are a Netra X1 I want to prepare and send to Germany for hosting, and a Cisco MC3810 that was supposed to do VoIP FXS for us so we could have an Australian phone number months ago, but I fail.
sakima, foster, eltoro
On top of foster is eltoro, a small desktop PeeCee running FreeBSD.
You can see part of sakima on the left.
sakima, ariel, setient's box
ariel is on top of that. ariel should replace ezln soon, and consume less electricity.
have a look at sakima, the 1U machine with no cover. It's power supply is bad, so it's running off the one in the foreground. something you can only do at home, not in a proper data center. I think it's fun.
sakima is running CentOS. sakima is one of the oldest machines on the Internet.
look closely, and you can see a tiny USB key fob disk plugged into it. This is because of a rather prickly problem with SVM and the quorum rules of metadb ``state database replicas''. Does this sound complicated? The rule is, ``you must have more than half of them available.'' And they give you two disks. so it's impossible.
Here is what the shelf used to look like.