Like many, I freaked out this week when I read that Verizon admitted to throttling connections from Netflix.  But then Verizon denied the rumor. Maybe the best answer is to keep a close eye on download speeds from different points on the net.

I’ve just switched home ISPs from to Astound. I loved Sonic’s politics, and their customer service was great. However, the fastest I ever got from them was 18Mb/s down, and about 2Mb/s up. For the same price, Astound gets me much better speeds, and as far as I can tell, Astound’s politics aren’t horrible. They’re not AT&T, anyway.

Server d/l speed, Mb/s
AWS, East 67
AWS, CA 100
AWS, OR 90
Linode, NJ 61
Linode, GA 52
Linode, TX 63
Linode, CA 80

Speed is very closely connected to geography, and that seems reasonable.

I think there’s room for a daemon app that runs a few times per week against a huge database of sites. Basically crowdsource speeds from all over the net. The app should fire at random times, and grab a site or two randomly from the database, then get a speed check. It would then report the speed , the ISP, the protocol, and the IP address of the checker with a timestamp to a central database. This would provide a crowdsourced monitor of what ISPs are doing with our packets.

Taking clean water to the desert is necessary, but then what do you do with the gray water from showers, dishwashing, etc? You can put it in tanks and bring it home, but that’s heavy and messy. There are pump-out services, but I consider that cheating.


Evaporate! P and I designed these towers in early August, and with some help, I assembled them on the playa. These towers are made of 7′ deer fencing and rebar, guyed to the truck with climbing webbing. They sit in kiddie pools, and a 12v bilge pump in each pool drives water through a hose to the tower’s top and over a snow saucer. Power for the pumps came from 110v AC provided by Camp Soft Landing and 12v adapters I got from ebay. The water runs down the sides of the tower, soaking the burlap. Sun and wind pull the water off the burlap. Excess water runs back into the kiddie pool to make another round.

evap-2013-trucktopNote the handsome blinkies (75 12mm WS2801’s from Adafruit) on the guy lines. I used Funkboxing’s LED effects with a bit of tweaking. They looked great!

These systems took care of gray water for the Full Circle Tea House and about 150 BM participants. The average was 20-22 gallons per day per tower, for a total of about 205 gallons. That’s over 1600 lbs.

evap-2013This year worked a lot better than last year. One key finding is that the kiddie pools are huge, and with them the reservoirs on the bottom, it takes a lot less attention to keep the reservoirs full compared to last year when we used a 5gal bucket as the reservoir. The combination of full-time pumping and the taller tower led to a 30% increase in daily evap rates. Good stuff.


This video is an example of my questions on this thread at the Adafruit forum.

The power and ground wires are getting set up in the blinky coat.

blinky-coat-power-bus-sewingHere the conductive thread is sewn through the coat, looping a pad from each NeoPixel to the appropriate power or ground wire. As noted in an earlier post, the power and ground wires have little cuts in the insulation so the conductive thread can make the contact. Here’s an overview of how things are as of yesterday:

More to come!

As in last year’s model, the LEDs draw 60mA (at 5V) per pixel. I’m now planning 76 pixels, so that’s about 4.5A — if they were all on bright white at the same time, which is rare. Nonetheless, that’s a lot of power.

I’ll bring the power from the battery to the pixels in buses, that is, relatively thicker wires (22AWG) that can handle the current. The positive (red) and ground (black) have little cuts in the insulation through which the conductive thread wraps each pixel to each bus. Here’s an image of the buses:

blinky-power-bus-markedAfter I marked them, I used a wire stripper to make a little break at each mark. Then I separated the sections of insulation just a tiny bit, lining them up, so the pixels can be wired into them.

The buses connect to the battery through voltage regulators that bring the power from whatever it is at the battery (7.2V in the batteries I’ll use in the coats, but 8.4V in the battery I’m using for testing) to stable 5V that the LEDs need. The FLORA can take whatever voltage we give it. The wiring from the battery looks like this.

blinky-power-harnessStarting on the far right, the molex plug connects to a battery. The negative (black) wire goes to the wire snap on the top; the wire snap connects all the wires in the snap: the battery ground, the ground wires for the input for the 3 regulators, and the ground for the JST that powers the FLORA. The red wire for the molex connects to the positive for the 3 regulators, and the positive for the FLORA.

The regulators each handle about 1.5A, so we use three of them. We’ll bring the positive from each regulator to a subset of the LEDs. The ground is common for all of them. The 4-pin JST on the bottom left will connect to wires that carry the current to the buses on the back of the coat.

Here’s an overview:

Last year’s evapotrons got rid of a lot of grey water, on the order of 175 gallons. Good, but I think the design suffered from the Arduino (from my code, anyway). The machine crashed a lot and got into weird loops, so the control devices I built ended up leaving the pump not running for chunks of time.

New attempt: we’ll just let the pumps run all the time. They’re not loud, and they don’t draw all that much power (about 25w each). No Arduino.

Also we realized that without the Arduino, the evaporation towers can sit in a single kiddie pool (we don’t need the two-stage mechanism we had last year). That means each tower can be made of deer wire and rebar. Here’s the structure:


In the image, you can see the deer wire formed in a 7′ tall, 36″ diameter cylinder.  At three equilateral points, a piece of 8′ rebar will be fixed to the deer wire to give it more vertical rigidity. The cylinder will be wrapped in burlap.


The rebar comes out of the cylinder at the top. From each rebar top, we’ll fix a line of webbing down to the truck, kind of like the diagram below.


Last year we used a strap of webbing wrapped around the truck to anchor the evapotron to the truck. This year, we’ll have 3 straps around the truck. Each evapotron will attach to the straps by 3 guy-ropes: one toward the center, and two to the front or rear corners. The guy ropes are 1000-lbs test ratcheted tie-downs, so I think they’ll hold fine. The guy-ropes also provide a way to level the top of the evapotron, which turns out to be Very Important.

We’ve also improved the water distribution mechanism on top of the saucer. Note the PVC is all fitted and glued.

evap-top-testUnderneath the saucer, the PVC attaches to a simple hose that drops to the pump sitting in the kiddie pool. No Arduino, though I may yet add an Arduino to drive some blinkies to show off the evapotrons. More to come.

I had kind of given up on last year’s attempt to make a coat covered in swirly bright LEDs. We made the coats, but they were way, way, way too heavy to be usable. The coats themselves were pleather & (fake) fur, so they were hot. Then 200 pixels, a 6-lbs battery, and very fragile wiring made them unusable. 

Try again! Adafruit’s NeoPixels are sewable and designed for wearable applications. I got to thinking: hmm, what if only 60 pixels? What if the pixels were all together (making layout simpler)? What if we just run super-simple patterns? Ok, back at it. Image

I found an Australian cattle-driver’s coat at a school rummage sale in Vermont, and my friend the experimental seamstress laid out a nice pattern (see photo and diagram).  


More images as we finalize the power and ground bus and the wiring. 


More BIOS stuff, but this was a lot easier than the previous update because the Acer D270, for it’s many warts, boots easily from USB. I used unetbootin to create a FreeDOS install. I copied Acer’s BIOS update (DOS version) to the usb, and booted the D270. The notes on this page told me to look in “drives” B: and C: , and indeed, the driver turned up on “drive” C: . I ran it, and the BIOS updated smoothly. 

Now if only I could get a linux distro running on the machine. The backlight has an ocean of troubles with the current kernel. In short, every time the machine boots or comes back from suspend, the backlight resets to a level too low to read. In essence, the screen goes dead. If you can feel your way to a terminal, you can get it back with this command (as root): 

setpci -s 00:02.0 F4.B=00

That’s irritating. As I fiddled with GRUB boot options (acpi_osi=Linux, acpi_backlight=vendor), xorg framebuffer settings, and drivers for various additional video bits, I broke the install and had to start over. How many times in my linux-life has this happened? Dozens? 

I finally got it working by downgrading Ubuntu to 12.04, and it Just Worked. That’s very unhelpful. But even then, I wasn’t all that close to a solution for a really working portable. The problem, as has often been true in my linux-life, was wifi.

The chip is a Broadcomm 4313, which should be well-supported under linux. And out of the box (with Ubuntu 12.04), the machine connected happily to the wifi at my house (with WPA2 at 2.4GHz) and my office (ditto). 

However, this machine was destined for a less-techie colleague. She needs to move around and have the machine connect to networks all over the place. And it didn’t: it wouldn’t connect to wifi at her hotel, at Moe’s house, or at cafes that require a web-based authentication to their router after DHCP. 

I tried any number of solutions: compiled the driver from Broadcomm’s published code; tweaked and twiddled the network interface; poked and prodded at iwconfig, iwlist, and their kin; spent a few hours watching the syslog as the chip tried to make DCHP connections; downgraded from the default driver to the b43 legacy drivers. And so forth.

Several of the tweaks changed the error pattern. We were able to connect to some networks with some combinations of drivers, and not with others. We got it going for a while on nearly all networks, then it stopped and wouldn’t work with any, with nothing helpful in the logs. 

Time is money, and I’d spent a hell of a lot on this little machine. It’s going back on the shelf, destined to be used only at Burning Man (where I make network connections using a cable at Cheddar’s camp). Moe took our colleague to the Apple Store and bought her an Air. Expensive, but enough with the fiddling already. We had her configured for LaTeX, R, RStudio, SublimeText, ssh, svn, make, and python (and all relevant packages for those tools) in ❤ hours. Count this as a linux fail. 

Media storage

I managed to get all my media collated onto a single set of disks. There are two 3TB RAID-1 disks, and I have a backup on a couple of other disks. Still, this all feels pretty fragile. I’m going to need to figure out a richer backup strategy.

It totals 2.2TB of heavily duplicated data (still, this is a lot less than the 6TB I thought it was). However, there are a couple of disappointing holes. The combination of bad interaction between Symantec’s PGP WholeDiskEncryption and Apple’s TimeMachine, and Apple MigrationAssistant’s occasional loss of local mail storage, has led to a couple of big holes in my archiving. In particular, I’m missing nearly all of 2009’s mail; various recovery strategies are underway. My archives from the DOS years 1985-1999, and the gnu/linux years 2000-2008 are a lot better than the OSX years 2009-present. All my actual work files (and their history) 2004-present are safe in our SVN repository, but still, I hate to lose even a single byte.

The failure rates are interesting as well. One of the 18 hard disks, one (a 2.5″ drive pulled from a laptop c. 2005) had a serious hardware problem. Of the 53 zip disks, all but 4 had I/O errors; but the 67 CDs had only 2 I/O probs. The 157 1.44MB 3.5″ floppies were in between with 22 damaged ones. ddrescue is of course the right answer, but if you lose even a few bytes of a pgp-encrypted file, or a bzip2, gzip, or zip compressed file, that data is lost.

Total so far: 7.5M files in 900K directories. However, this includes zillions of zip and tar files which I need to unpack before indexing. More to come.

I have an old linux server based on an Abit KN9-SLI motherboard. This is an ancient machine, circa 2006, and it’s seriously creaky; it has only 2GB non-ECC RAM, and that’s not enough.

I want to use it as the host for an RAID-1 array of hard disks where I’ll accumulate all of the files I’ve ever had. Literally. It’s a LOT of data, about 6 TB, but there’s tons of duplication. Dealing with duplicates is another question for later.

First I need a machine with adequate memory to handle the processing I want to do; the existing 2GB of non-ECC isn’t enough. The motherboard takes DD2, and I found cheap 2G unbuffered ECC DIMMs here.

When I put them into the slots on the mobo, it failed to post to the BIOS; it just gave plaintive slow beeps, which indicate RAM trouble. I pulled two DIMMs out, and it did boot. Hmm, maybe the BIOS only knows how to handle 4G? Furthermore, I found that linux wasn’t recognizing the ECC.

A BIOS update was in order. However, Abit is out of business, and their website has dead links for the actual files. There were links in lots of dodgy places, and I finally found this guy who archived all of Abit’s BIOS updates.  Ok, now how to flash it?

The server in question doesn’t have an internal floppy drive, and the CDROM is flaky. I struggled down many blind alleys and wasted a few hours, but finally found excellent instructions on the Ubuntu forums. I created a 1.44M, 3.5″ floppy disk, and booted. Wow, it’s weird to boot into DOS (using an external USB floppy):


I ran the BAT file (my head spins), got a little worried as part of the instructions said that I should check my “floopy” [sic] drive, but I went ahead and flashed the ROM:


Following the instructions in the mobo’s Fine Manual, I reset the CMOS, then added the RAM (so there were all 4 DIMMs = 8G), booted to post, updated all the settings in BIOS, and it worked!

wylbur@snowball:~$ sudo dmidecode
# dmidecode 2.11
BIOS Information
   Vendor: Phoenix Technologies, LTD
   Version: 6.00 PG
   Release Date: 09/04/2007 <---this is the update
Base Board Information
   Product Name: KN9(NF-MCP55 series)
   Version: 1.x
[ 6.640164] EDAC MC: Ver: 2.1.0
[ 6.652770] MCE: In-kernel MCE decoding enabled.
[ 6.653797] AMD64 EDAC driver v3.4.0
[ 6.653833] EDAC amd64: DRAM ECC enabled.   <--- YAY! 
[ 6.653838] EDAC amd64: K8 revF or later detected (node 0).
[ 6.653853] EDAC MC: DCT0 chip selects:
[ 6.653855] EDAC amd64: MC: 0: 4096MB 1: 4096MB
[ 6.653857] EDAC amd64: MC: 2: 4096MB 3: 4096MB
[ 6.653859] EDAC amd64: MC: 4: 0MB 5: 0MB
[ 6.653861] EDAC amd64: MC: 6: 0MB 7: 0MB
[ 6.653882] EDAC amd64: CS0: Unbuffered DDR2 RAM
[ 6.653884] EDAC amd64: CS1: Unbuffered DDR2 RAM
[ 6.653885] EDAC amd64: CS2: Unbuffered DDR2 RAM
[ 6.653887] EDAC amd64: CS3: Unbuffered DDR2 RAM
[ 6.653932] EDAC MC0: Giving out device to 'amd64_edac' 'K8': DEV
---and here it is-----
wylbur@snowball:~$ free
 total used free shared buffers
Mem: 8200936 396520 7804416 0 14852
-/+ buffers/cache: 187188 8013748
Swap: 3903788 0 390378

It will run memtest for a day or so, and then back to hammering away at the giant pile of disks. I’ll document that process in a subsequent post.

(Also playing with MarsEdit blog editing software. Current sense: works pretty well, nice clean interface)