I have an old linux server based on an Abit KN9-SLI motherboard. This is an ancient machine, circa 2006, and it’s seriously creaky; it has only 2GB non-ECC RAM, and that’s not enough.

I want to use it as the host for an RAID-1 array of hard disks where I’ll accumulate all of the files I’ve ever had. Literally. It’s a LOT of data, about 6 TB, but there’s tons of duplication. Dealing with duplicates is another question for later.

First I need a machine with adequate memory to handle the processing I want to do; the existing 2GB of non-ECC isn’t enough. The motherboard takes DD2, and I found cheap 2G unbuffered ECC DIMMs here.

When I put them into the slots on the mobo, it failed to post to the BIOS; it just gave plaintive slow beeps, which indicate RAM trouble. I pulled two DIMMs out, and it did boot. Hmm, maybe the BIOS only knows how to handle 4G? Furthermore, I found that linux wasn’t recognizing the ECC.

A BIOS update was in order. However, Abit is out of business, and their website has dead links for the actual files. There were links in lots of dodgy places, and I finally found this guy who archived all of Abit’s BIOS updates.  Ok, now how to flash it?

The server in question doesn’t have an internal floppy drive, and the CDROM is flaky. I struggled down many blind alleys and wasted a few hours, but finally found excellent instructions on the Ubuntu forums. I created a 1.44M, 3.5″ floppy disk, and booted. Wow, it’s weird to boot into DOS (using an external USB floppy):

Photo

I ran the BAT file (my head spins), got a little worried as part of the instructions said that I should check my “floopy” [sic] drive, but I went ahead and flashed the ROM:

Photo

Following the instructions in the mobo’s Fine Manual, I reset the CMOS, then added the RAM (so there were all 4 DIMMs = 8G), booted to post, updated all the settings in BIOS, and it worked!

wylbur@snowball:~$ sudo dmidecode
# dmidecode 2.11
<snip>
BIOS Information
   Vendor: Phoenix Technologies, LTD
   Version: 6.00 PG
   Release Date: 09/04/2007 <---this is the update
<snip>
Base Board Information
   Manufacturer: http://www.abit.com.tw/
   Product Name: KN9(NF-MCP55 series)
   Version: 1.x
---dmesg-----
[ 6.640164] EDAC MC: Ver: 2.1.0
[ 6.652770] MCE: In-kernel MCE decoding enabled.
[ 6.653797] AMD64 EDAC driver v3.4.0
[ 6.653833] EDAC amd64: DRAM ECC enabled.   <--- YAY! 
[ 6.653838] EDAC amd64: K8 revF or later detected (node 0).
[ 6.653853] EDAC MC: DCT0 chip selects:
[ 6.653855] EDAC amd64: MC: 0: 4096MB 1: 4096MB
[ 6.653857] EDAC amd64: MC: 2: 4096MB 3: 4096MB
[ 6.653859] EDAC amd64: MC: 4: 0MB 5: 0MB
[ 6.653861] EDAC amd64: MC: 6: 0MB 7: 0MB
[ 6.653882] EDAC amd64: CS0: Unbuffered DDR2 RAM
[ 6.653884] EDAC amd64: CS1: Unbuffered DDR2 RAM
[ 6.653885] EDAC amd64: CS2: Unbuffered DDR2 RAM
[ 6.653887] EDAC amd64: CS3: Unbuffered DDR2 RAM
[ 6.653932] EDAC MC0: Giving out device to 'amd64_edac' 'K8': DEV
0000:00:18.2
---and here it is-----
wylbur@snowball:~$ free
 total used free shared buffers
 cached
Mem: 8200936 396520 7804416 0 14852
194480
-/+ buffers/cache: 187188 8013748
Swap: 3903788 0 390378

It will run memtest for a day or so, and then back to hammering away at the giant pile of disks. I’ll document that process in a subsequent post.

(Also playing with MarsEdit blog editing software. Current sense: works pretty well, nice clean interface)

Image

wylbur with the evapotron on top of the truck at Camp Above the Limit.

Image

We put the evapotrons on top of trucks because up there they are exposed to sun and wind all the time. No shadows as the day gets longer. The smell and mist of grey water isn’t so nice in a crowded camp, but on top of the truck, no one notices. The truck’s roof also serves as a final catching point for overflow; splashing grey water evaporates from the top of the truck instead of getting on the playa. Finally, the location up a ladder discourages random people from dumping grey water into the system, which was a problem in 2011.

Here are the evapotrons in action at Burning Man. We eliminated over 1500 lbs of grey water. Oh yea!

The evapotron on the truck at Coffee, Tea and Me

Close up of evapotron at Coffee, Tea, and Me

 

View of evapotron at Above the Limit

 

The weight sensor was an interesting little beast. I’ll write a long post at some point about how hard it is to figure out with a sensor how much water is in a bucket. I tried the eTape liquid sensor, various electrical sensors in the water, ultrasonic measurement of the height, and finally a weight sensor. The final one was the only one that worked at all — and it didn’t work all that well. More on that later.

However, it did sort-of work, and that can be seen here in the data between 12:45pm and about 3pm on Sunday 26 Aug. The cycling from low level to a high level shows the pump’s action. When the pump is running, most of the water is up on the evaporation tower, not in the reservoir bucket, and so the weight is low. The pump shuts off, the water drains back into the bucket, and the weight goes up.

Not *all* the water drains back. In the graph above, starting at about 13:30, imagine a straight line through the tops of the line’s plateaus. That line is tracking the loss of water due to evaporation off the tower.

There are tons of complicating factors: water is added to the system, as shown in the previous post; the measured weight is noisy because the truck (on which the whole tower sits) is jostled by wind, as people move stuff around inside, and when they climb onto the truck; and occasionally the sensor reports completely nonsensical readings. I’m working on an analysis that will control all these effects to get at evaporation rates.

Figuring out how much grey water went through this machine was pretty complicated. We kept a logbook, but I worried that it would be honored “more in the breach than in the observance,” as my friend Cheddar said. He was right. Image

This graph shows the cumulative number of pulses recorded by a flow sensor attached to the filter bucket. All the grey water put into the evapotron went through this sensor. One key finding from this graph is that there are many, many more upward steps in the graph than records in the logbook. People put grey water into the system frequently without recording it. Cheddar worried about this on the last day (when a LOT of grey water went in, starting just after midnight on Sunday 2 Sep: what were people doing putting grey water into the evapotron a couple of hours after the Burn?). However, it seems that people put the odd one or two gallons in throughout the week.

In tests before I built the evapotron, I found that the flow sensor is extremely imprecise. Although it is rated at 192 pulses/liter, in tests I recorded from 160-240 pulses/liter. Given a final count of 93395 pulses, that suggests 128 gallons, in a possible range of 102-153 gallons.

This estimate probably overstates the total because there was about 8 gallons in the system that had to be brought home at the end. Still, given that it ran for 8.5 days, this represents 15 gallons/day (in a range of 12-18 gallons/day).

And there was a second evapotron running. It didn’t do as well, but it handled another 70 or so gallons. Between the two evapotrons, I estimate that we eliminated three-quarters of a ton of grey water. I’m calling this one a success, though there are many lessons learned and better stuff to build for next year.

ImageThe charging circuit worked pretty well. The idea here is that there are two batteries. One is connected to the bus which runs the Arduino, sensors, and pump. The other is connected to the AC charger. The Arduino controls relays which switch the batteries between the two. The upper line in the graph shows the voltage in the bus circuit. When the line is red, battery A is driving the bus. Each time a battery is switched into the bus circuit, it starts with about 13v, then voltage drops as it discharges.

What this graph shows is that the two circuits worked together. In each cycle, the charging circuit (on the bottom) got through the first phase (the slow ramp), then the second phase (the plateau at 14.8 volts), then the third trickle phase (the plateau at about 13.6 volts). When the bus circuit fell below 11 volts, the Arduino flipped the relays, reversing the batteries so the discharged one could charge, and vice-versa.

 

It seems like it’s actually working in this 45Ks (nearly 13 hours) run:

This is Box B, so it’s nice to see that it’s doing ok. The point here is that the batteries are handing off to each other smoothly. Note that if the bus battery runs out of power, it does a bunch of very short pumping cycles (see from 20-23Ks). This isn’t very effective, I don’t think, but it’s still probably the preferred behavior.

One slightly worrisome note: I’m clearly not sensing the charging cycle correctly. Note that the charging graph climbs, holds the peak, then drops to a plateau for a while (3600 seconds, actually) before handing off to the other battery. The handoff is being triggered by logic that notes a long plateau in the charging voltage. Somehow the logic that spots the peak and descent between phases 2 and 3 isn’t working. So time is being wasted while the plateau-sensing code waits to make sure this is a real shift to charging phase 3.

Maybe I should sense the slope of the charging cycle? Or just shorten the time the plateau is being watched? Dunno, but either way, the charging cycle can be shortened by at least 1800 secs, maybe even 3000s.

That’s it, testing time is exhausted. Time to go to the playa.

Most things are working properly:

The battery switching looks good. However, the pump hung at time==14617. At that point, for its own reasons and without input, the flow sensor decided that the flow rate is infinite. There’s a bug in it’s calculations, but whatever, at this point I have to ignore the flowrate calculation in the code from Adafruit, and monitor the flow sensor’s pulse count directly.

I did a six hour test run of the evapotron control circuit today, with mostly positive results.

This graph shows the bus voltage, the depth in the reservoir, the current drawn by the pump (in mA), and the voltage in the circuit being charged. It all makes sense, but it might benefit from a walk-through.

The bus voltage declines over time, of course, because this is the battery being drawn on to run the system. Periodically it drops about 0.8v — which is when the pump switches on. The depth stays roughly constant at about 25cm of water in the reservoir, which I’m measuring by using a load sensor. The pump can push water up faster than it returns, so occasionally the pump doesn’t finish it’s cycle before it runs out of water; those are the interrupted cycles. The current in the third panel is the inverse of the bus voltage and depth curves: it’s high when the pump’s running, and otherwise just noise and leakage (this is a little troubling but may just be sensor noise).

Then the charging circuit reminds us how three-stage lead-acid battery charging works. I’m watching it for when it drops back to 13.75v, which is when it hits stage 3. At that point, the battery on the bus could hand off to the newly charged battery, and charge itself.

This cycle doesn’t work the pump hard enough. I’m off until Monday, but I’ve got a couple of days to tune the work-charging cycle to find the max time the pump can run matched to the battery charge time.

Oh, and the second evapotron? Not done, but it’s close. The data logging shield for the Arduino doesn’t work (we’re debugging on the Adafruit forums); the sensor cables aren’t made, and the whole thing isn’t tested. I wish I were substantially farther along, but it’s all going to have to be enough.

(I have a long blog rant coming about how hard it is to measure liquid depth — the eTape failed; the Ping))) failed; and various attempts to test resistance of the water all failed)

if banshees were into hobbyist electronics, anyway.

Hard to see in the blurry photo, but this is the third coat, running. The three Otter-boxes housing the Arduinos are on the mat (only one is working, the other two have their signal and power cables flopped out).

It’s been a long road, and I still don’t have software or a control system. Each control box has a 4-bit dip switch to choose alternative programs, and a potentiometer knob for varying the refresh/flash speed. However, I’ve written software for neither.

I dumped a strand and Arduino on my friend Slobberchops, he’s an Alpha Geek. I’m hoping he gets interested enough to write me some cool patterns in the 10 days before I leave for the playa. Holy cats, only ten days; I’m not panicking, breathing, calm.