Pictures

All our bills are paid now. I still don't want to brag about this stuff because, honestly, although maybe it's neat, it's a generation behind. We're failing to do certain things that should be done with this number of computers:

• virtual storage. We should have some big disk array that has, if not actual backups, at least snapshots and failed-disk-protection. All the machines should take LUN's out of the big array. The big array should have disk encryption, and should require manual keying if it loses power. I believe pretty much every site should be built around something like this.
• power saving. Each person gets his own box, which is some piece of junk he found abandoned in a hallway or bought on eBay for $100. The old computers don't exactly guzzle power, but they are so slow they could be combined into VM's on one fast computer that uses less power than ten slow ones. or we could buy low-power boards like Via C7 (space.net does this. too bad their prices are so insulting.).
• proper UPS with a bypass switch. Our UPS goes out more often than ConEd. For it to actually help rather than harm overall reliability, we should have a bypass switch so we can maintain the UPS without shutting down all the computers. As is, when I changed the batteries I soldered one DC lead at a time with an ungrounded butane iron and try not to touch anything metal and grounded while I'm doing it. We should also have a way to load-test the UPS with a space heater or something before throwing the bypass switch back to the normal position. And maybe we should have two UPS's for machines that have two power supplies. We do have already two separate clean power circuits wired back to the breaker panel.
• better Internet connection. T1 is slow and expensive. There doesn't seem to be anything cheaper for the same amount of upstream bandwidth, though. We are thinking of getting a worse Internet connection soon (half as much upstream bandwidth).

Once we fix all those things, we could get a generation ahead, and set up some kind of liquid cooling manifold system. If we can run the equipment hot enough to keep the effluent temperature above the outdoor temperature, we can save some money on aircon that way.

Also, in the winter it gets hot in here, and we have to open windows which I think makes the overall apartment building less sealed, and wastes oil. What I mean by this: forget the computers. I think it would help if we built a hypothetical cooling system for winter when it's close to freezing outside, one with circulating water/antifreeze, a coil outdoors, a coil indoors, and fans on both coils. This way, we could cool just our apartment rather than the building, and do it without opening windows. When we open windows, warm air blows out of our windows, and cold air blows in through the other side of the building. Consequently, people on the other side of the building open up their radiator valves, turn up the whole-building thermostat, and so on, using more oil. If we could keep our windows closed, the overall building would be slightly more sealed, and less outdoor air would blow into the opposite-side apartments where people are too cold.

In any case, it sucks in here during the winter. If we don't open windows, it's above eighty, all winter long. It's horrible. We only get relief when the furnace breaks. When that happens, it gets colder and colder by about ten degrees per day with no bottom in sight.

---

UPS and makeshift blower

The blower is made from a window aircon we grabbed from the East Village on recycling day. We took one with the orange sticker on it so the CFC's were already emptied. Then I removed the compressor, the coils, and the back fan. It sucks some of the air through that 6" tube from a 35,000 BTU window aircon in another room. I think we need an 8" tube instead. It is a ``high-pressure ducting'' system. :(

---

setient's box, foster, and old sakima.

I don't know the name of setient's box. I think it's a SPARCstation 4 running Solaris 8 or 9. foster is a fairly modern AMD PeeCee running Gentoo Linux. sakima is also a PeeCee, some kind of Dell, but is replaced with a much newer machine now running CentOS Linux. sakima is one of the oldest machines on the Internet. On top of sakima is eltoro, a small desktop PeeCee running FreeBSD.

---

most machines

In the pizzabox tower there's castrovalva, another shell server and a hackint.org irc server. castrovalva is a Digital DS10 running an old version of NetBSD with horrible pthread bugs. There's antioch, another shell server, running I think NetBSD on a SPARCstation 10. ``shell'' means web, mail, DNS, 'screen' sessions, a working C compiler, things like that from the old Internet. Above these are a 100BaseTX hub (an actual hub, as in 10Mbit/s devices don't work) and a couple very cheap, junky switches.

To the right are the two fishsticks, which are full of disk drives. They serve the contents of the disks over SMB and do little else. They run Gentoo Linux. The one on the right netboots off the one on the left using a custom 'genkernel' script. We got a cheap tower case marketed for loading up with seven CD-ROM's (SCSI lets you have seven, remember?). I'm surprised they still make cases for this niche, but it turns out to be the perfect cheap way to mount disks, in my opinion, and give them proper ventillation. Each disk goes into a 3.5"-to-5.25" bracket and has a vented metal faceplate in front of it. In one of them we dremmeled out the top of the case and added a filtered, high-flow (~$10) fan from Allied. They're running Gentoo Linux, which I hate, but BSD doesn't have any filesystems I feel are appropriate for large volume sizes. Each disk has a separate filesystem on it, no RAID. This no-RAID is a carefully-thought-out decision which I will defend pretty vigorously.

Underneath the fishsticks is terabithia, a Sun 280R that WANg gave me. Unfortunately it's broken. I'm going to buy another one like it which will also be named terabithia. This will replace castrovalva with something that runs Solaris and has a bunch of things NetBSD doesn't, such as a proper large-disk filesystem, working pthreads, zones, an automounter that doesn't panic the kernel, an iSCSI initiator, a C compiler that's performant on non-i386 CPU's, working thread-aware debuggers for both userland and kernel, and ability to run compiled languages other than C, such as Java and Lisp. I can't wait to get that done.

To the left is ezln, a Digital 'avanti' (PWS 433au) which is a NetBSD PF firewall, doing HFSC with some PF/ALTQ patches which after a couple years I think are still not integrated into NetBSD. I will replace it with some FreeBSD/sparc64 box as soon as I can---so sick of this NetBSD shit (but NetBSD 5.0 is going to be awesome. no, I'm not joking. It will be! but it still won't have device polling, so for the router, FreeBSD it is, forever.). On top of ezln is a C1720, the T1 router. I had to buy more memory for it so it could run an IPv6 IOS image.

Not pictured is lucette, a Sun Netra T1 200 colocated at Pilosoft on 55 Broad St. It has a 70W power supply, proper 1U rails, two small, reliable SCSI disks, and a Broadcom 57xx chip which has a good driver in FreeBSD/sparc64, which is what it runs. It runs PF/HFSC. FreeBSD has device polling. This filters and schedules our traffic before it enters the T1, does NAT, and maintains IPv6 tunnels and tinc tunnels for KrautVPN. It doesn't run any web servers or Xvnc or any of that nonsense---we do all that from the apartment where we have an arbitrarily large amount of disk space.

Here is what the shelf used to look like.