Is Python an Acceptable Haskell?

So I struggled with some Python this weekend. I'm taking over a piece of code that is written in Python, but the original is very imperative, with a lot of for loops. It was giving me a headache. It just looked way too long and tedious. I'm too old to spend my life debugging off-by-one errors in for loops.

After staying up half the night struggling with a malfunctioning IDLE on MacOS X, I finally discovered the cure for the headache was -- more Python!

# This gives us a "curry" equivalent
from functools import partial

# Reduce a list of strings to a single string by concatenation
reduce_stringlist = partial( reduce, str.__add__ )

# Given a list of XML elements, return a list of only the data from only the text nodes
# (ignores any non-text nodes)
def get_text_data( nodelist ):
    return map( lambda node: node.data,
           filter( lambda node: node.nodeType == node.TEXT_NODE, nodelist ) )

# Now, combined: reduce a list of nodes to the concatenated text extracted out of only
# the text node data
def extract_text( nodelist ):
    return reduce_stringlist( get_text_data( nodelist ) )

# Given a portion of the parsed XML tree and a name, extract a single string. Expect
# only one element.
def get_one_text_element( node, name ):
    elt_list = node.getElementsByTagName( name )
    assert elt_list.length == 1
    return extract_text( elt_list[0].childNodes )

# Given a starting node and a list of tag names, retrieve one text string each and
# put them in a newly created dictionary using the tag name as a key
def make_text_element_dict( node, namelist ):
    return dict
        zip( namelist,
             map ( partial( get_one_text_element, node ),
                   namelist ) ) )

Ahhh, I feel much better! All that excess hairy boilerplate seems to be just melting away!

The only thing I did not like is that there did not seem to be a nice way to specify keyword arguments to "reduce" so that I could create a partial (curried) application that supplied the first and optional third parameter, leaving the second as the one to be supplied at runtime.

I experimented with writing the above with Python's list comprehension idiom, generators, and various permutations on join(). The above just seems to make more sense to me. Haskell has apparently ruined me for other languages.

Apparently Guido would like to ban reduce() from Python. I say -- prefer the standard idiom to the offbeat, and avoid the Not Invented Here syndrome. Python's comprehensions and generators are nice, but apparently I've been ruined by seeking more and more expressive languages, seeking truth and beauty on my wandering but inexorable path from Dylan to NewtonScript to Scheme to Haskell. Lambda, map, curry, reduce, and zip now seem to me to be fundamental primitives that any reasonable dynamic language ought to provide, and it seems to me that they ought to be preferred to an obscure language-specific trick. As long as I have to use Python, Guido will have to pry the curried functions from my cold, dead hands!

A Pet Peeve -- Proprietary Fasteners

As if it wasn't hard enough to keep tools on hand for the following screws:

- Flat blade

- Hex (in metric and fractional inch sizes)

- Philips

- Torx

- Tamper-resistant Torx (with a post in the middle)

- Torx Plus (with wider holes)

- Tamper-resistant Torx Plus (hard to come by -- with wider holes, a post in the middle, and five points instead of six)

- TTAP (a six-point variation with a deep hole in the middle, also hard to come by)

- Tamper-resistant TTAP (which I've never seen)

- Several other unusual fastener heads, such as square, two-pin, etc.

I've found that certain hard drive screws use a five-pointed variant of the six-pointed Torx star shape, which corresponds to "none of the above" and which doesn't seem to be available, anywhere. I found this out the hard way after buying a very nice set of Torx bits. The screws in question are so small it is very hard for my 40-year-old eyes to distinguish the five-pointed star from the six. They are roughly the size of a Torx T3 or T4.

The only reason I can imagine that manufacturers would use a fastener like this is sheer perversity. It isn't to make the devices tamper-proof -- the screws can be turned with a flat-head screwdriver of just the right width, although this is not ideal for either fastener or tool. Or, I could always just drill them out, as I've done on more than one occasion with a very stripped or damaged screw.

To the best of my knowledge the correct tool is not available for purchase _anywhere_ for a nobody like me, although I think you can sometimes find tools for larger versions of the bits for sale on eBay.

Several of these fasteners are apparently actually patented, and several apparently can't be legally sold except to properly licensed OEMs or other authorized personnel. Wiha sells, but will not sell to me, tamper-resistant Torx Plus tools, for example. And even if they would, they don't seem to have any this small.

As far as I'm concerned, such restrictions should be illegal. I'm sure there is an analogy here to be made to software APIs. I'm reminded of a talk I saw by Stallman, in which he drew puzzle pieces representing APIs, and talked about the evils of proprietary APIs. I bought it, I ought to have the right to take it apart!

By the way, the device in question is an iPod hard drive, made for Apple by Toshiba. But I'm sure there are many other variants of the same thing going on.

Trying Programming Environments for Children

I am home sick today, feeling terrible and feverish, and should be sleeping, but can't sleep, so I thought I would try out some of the programming environments for kids. A while back I set up my son with the Haskell tools used for The Haskell School of Expression and Thompson's Craft of Functional Programming book, hoping my son would work his way into one of the texts a bit. He didn't get that far, so I thought I'd try out some other options for him.

I started learning when personal computers came with BASIC interpreters built-in, so I am looking for something equally as easy to start with, although I think teaching my son BASIC might be considered child abuse.

I thought I'd try to set him up with Hackety Hack, a Ruby environment found here. On Ubuntu Gutsy, I found that it segfaults immediately upon launch. According to the forum this issue has been reported for close to a year, with no patch.

OK, the next one on my list to try is Greenfoot. Greenfoot is Java-based, and requires a JDK. I haven't been a Java hacker since the dot-com crash, but at one point knew it pretty well, so how hard could it be for a feverish software engineer? My install of Gutsy tells me its java is "java version "1.7.0, IcedTea Runtime Environment (build 1.7.0-b21), IcedTea Client VM (build 1.7.0-b21, mixed mode, sharing)." So I'll try installing the IcedTea JDK. That looks promising, although the notes indicating it is a "temporary fork" make me a little nervous; Ubuntu lives up to its ease-of-use repuation, the JDK installs, and the Greenfoot installer now seems to be able to figure out for itself where the JDK is. The Greenfoot executable launches, and looks at least moderately interesting, so I'll call that a qualified success for now, and see if he can get his head around it.

I had it on my list to check out Nodebox, but that's for MacOS X only, and today I'm on the Linux box, so we'll save that for another sick day.

Finally, I wanted to take a shot at setting up Sugar, the Python-based environment that was designed for the One Laptop Per Child system. I'm not hugely keen on Python; having used Python and Ruby, I harbor a slight preference for Ruby, and I've got a laundry list of things I don't like about both of them. However, I'm being asked to use and get comfortable with extending several tools in my workplace that are based on Python, so for better or worse I'm getting myself re-familiarized with Python. It's quite a mature language with lots of libraries. My son could do worse, so let's see what Sugar has to offer.

There are various ways to get Sugar going on Ubuntu. See the Wiki pages here. To begin, I'm going to try the emulated route, where my Linux PC will literally emulate an OLPC laptop. The other options look considerably more complicated and error-prone; in particular, there is an absolute rat's nest of required packages. So let's see how painful setting up emulation is. First, we download a bzip'ed image file. While it is downloading, I install QEMU, which is not difficult. For now I'm not going to attempt to install the kernel support for accelerating QEMU; we'll see if it is really painfully slow. This machine is "only" a Pentium 4 2.8 GHz system that I built from parts a few years ago; pretty obsolete, but usually plenty fast.

So, image is done downloading, unzip it, and try running QEMU on it -- it works! Although it does indeed take a very long time to launch. I still find it quite surreal to watch Linux boot on top of Linux!

That's all for now -- I am still home sick, after all -- it's time to have some hot lemon tea and try to get a nap!

Followup: installing the kqemu acceleration tools does make the OLPC image run _significantly_ faster -- it is quite tolerable now, although I have no idea how it compares to how the image runs on the target hardware itself.

I Voted. I Think. Probably. Maybe.

So, I don't know if you have been following news of Michigan's primaries, but they seem to be an utter farce this year. There is some kind of a dispute between the national Democratic party and the state party. The side effects are:

- Several of the biggest-name Dem candidates (Edwards and Obama among them) are not listed on the primary ballot.

- There's a write-in option, but if you write in one of these missing candidates, your vote will not be counted for that candidate(!)

- One of the candidates (Dodd) was listed, but he actually dropped out of the race earlier. (According to the results he still received 1% of the vote).

The recommendation was that if you wanted to vote for someone else, you should vote uncommitted. According to the results on Wikipedia, a whopping 40% followed this recommendation.

My precinct uses paper ballots with optical scanners. These are generally considered to be pretty reliable, and there is a paper trail in event of a recount. But this year, when I approached the scan machine, its counter read 466. After scanning my ballot, it still read 466. I asked the staff member about this -- she told me that it should read 466 and that if the machine took in the ballot and didn't spit it out, my vote was counted. Me, I'm not so sure. But the kids were in the car and my wife was late for a class so I didn't have time to stand there and try to determine exactly what was happening. But I left feeling somewhat uncertain that I had actually cast my vote. This just does not seem like democracy as it should be practiced!

A Bank Tries to Help Out

I got a call from Midwest Financial Credit Union -- it seems that someone in management read my blog post. We had a long and interesting conversation and I ended the conversation a much happier customer. For now, we are going to keep our accounts, and I am happy that they are doing some things to try to keep our business.

Among the details:

The hold on deposits was part of their anti-fraud measures in place automatically for the first 30 days of a new account. Apparently this is when they get hit with the most fraudulent deposits. A warning at the time we deposited the check would have helped us out.

The $100 limit of check card transactions applies only when using the card like a credit card, as opposed to a debit card. So I can still use it as a debit and make reasonable-sized purchases. The staff member I spoke to agreed that the $100 limit is unreasonable for any real-world use and is going to try to get it raised. It would have been really helpful to be warned at the time the cards were issued, or better, beforehand, that this limit was going to be in force.

So, why such a low limit?

The conversation was illuminating. In recent years I have put in some effort and a lot of money to try to clean up my credit report. This included starting payment on some debts that were dormant for many years.

The largest of these was a medical bill for a single night in the hospital. I was fully insured at the time, or so I believed; the hospital I presented my cards to told me that they accepted my insurance; but yet I would up owing somewhere north of $5,000 because my insurer did not want to pay what the hospital wanted to bill. The hospital was not one of the insurer's "preferred" providers.

Ultimately my insurance provider paid almost nothing, but also revealed to me the collusion that goes on between hospitals and insurers -- the individual line item fees the hospital charges to their supported insurers are in some cases as low as 10% of the same item as it is billed to someone with no health insurance. But that is a rant for another day.

I ignored this bill for many years more out of disgust and anger at the insurer and hospital than out of the inability to pay, but last year contacted the credit agency that owned the debt and started paying it down. It's now about 3/4 gone.

I also had the interesting experienced of getting sued for an ancient phone bill. The bill was real, but I had become so disgusted at AT&T's refusal to accept any payment plan other than immediate payment in full, and their dogpile of additional fees and charges, that I vowed they would not get a penny from me.

Well, they did, but not willingly. The debt was sold, and sold again. I wound up paying an attorney to come up with a settlement for me, and paid the settlement. It was again highly illuminating to find out that the agency in question completely ignored several written settlement offers that I tendered, but as soon as an attorney's letterhead was involved, settled for less than I had offered. I have a piece of paper in my file that says the bill is paid.

They say that no good deed goes unpunished, and it appears that settling one debt and nearly paying off another has damaged my credit rating, because both debts are now listed as currently "in dispute" instead of just hanging on as old debts. So, it is time to write some letters. There are procedures to go through to get items on a credit report corrected. But I refuse (again, on those damned principles of mine) to pay to receive my credit scores, so instead I will have to rely on the free annual reports I'm entitled to. Paying the rating agency for my scores seems too much like extortion, as if I had to pay eBay to improve my seller rating.

So, on balance, I'm much happier with the Credit Union, but much less happy with the creditors who are screwing with my life. And I have more crap to deal with. But I guess that's what you need to do in order to play the banking game.

Banks that Suck

Sorry, still no Haskell content. Please don't delete my blog! I'm working on it.

So, we were in the process of migrating our finances to a new bank, after our old bank, Republic, was pwned by Citizen's Bank and stopped doing all the things we liked about it, and started doing all the things we don't like (killing off our overdraft protection arrangement, stopping our various automatic repayment arrangements, removing access to our overdraft account from their online site, charging ludicrous fees, taking 3 days to clear electronic transactions, back-dating checks to try to hit is with more overdraft fees).

Based our on readings of their various rates and policies, we chose Midwest Financial Credit Union, here in Ann Arbor.

Less than a month later, we're now planning to close the accounts we just set up and continue our hunt for a decent bank that doesn't treat our accounts like an opportunity to slam us with fees at every opportunity -- just like Republic Bank didn't. I had my Republic account for about fifteen years and they helped me through many difficult times. Does such a place exist?

Midwest Financial got on our bad side immediately by taking ten calendar days to process a deposited $3,000 check -- while we were out of town. They finally managed to clear it (I think they took it to the originating bank in Erie, Pennsylvania by riding a mule along the old Erie Canal). It's in the fine print that they're allowed to do such things, apparently. (It's in the fine print that they can do just about anything they want, apparently).

Yesterday I tried to make a fairly large purchase (about $600) using a check card on the account -- it has a VISA logo). We have similar cards for our Citizen's Bank account and they function as either debit cards (with a PIN) or credit cards; they work fine, and we've never had an issue like this.

Anyway, the transaction was denied. Today I got word in my e-mail that a much smaller purchase (about $125) that I made online was also denied. Which is odd, because according to our most recent statement we had just shy of $3,000 in that account.

My wife went to the bank to talk to them and apparently someone's credit rating (possibly hers, since she opened the account) is rather low (we already knew that, thanks; that's old news, mostly from her days, now seven years gone, as an under-emplyed single mother), and therefore there is a $100 limit on transactions. Even though there is $3,000 in that account. Now, for a family of 5, $100 is not even a largish grocery store run. Some of these transactions could go through as debits, I guess, although there is probably some relatively low limit on the debit transactions as well. And we have a card with a VISA logo because not everyone is setup to handle PIN-based debit transactions.

They told her she can apply for a 48-hour waiver to make a large purchase, or in four months we can apply to have our "credit" limit raised.

This is all too much. We're going to do neither; we're going to fire this goddamn bank and get our money back. I'm giving serious thought to turning our money into gold, silver, and platinum bars and burying them in undisclosed locations! Maybe we'll just keep our old accounts; we're starting to get used to the exact ways they screw us, as opposed to all these new ways! We are very fortunate in that here in 2008 we are finally getting ourselves to the point where we have a little bit of a safety margin in our accounts, which kept us from getting stranded while we were out-of-state on vacation. But if we didn't have that margin -- if we were still living close to the edge -- we'd have been absolutely screwed. And having been that indebted slob for most of my life, I'm only going to do business with institutions whose policies are designed to be fair to that person, not to make their problems far worse.

The Potts Vacation 2007

So, we are back and the holiday trauma is over. We took the family to visit my cousins in the Washington, DC area and saw various friends along the way. I have just a touch of ranting I have to get off my chest before I can write anything else!

The trip, on Amtrak, went well for the most part. It is not easy, though, getting a disobedient pre-schooler, an infant, a teenager, two car seats, and a lot of unchecked luggage around.

The train from Toledo to Rockville, MD was only late getting into Toledo by about an hour. I was rather surprised!

We stayed in Gaithersburg with my cousin. We had planned to stay there each night, but things became more complicated. My cousin has just been diagnosed with a thyroid illness called Graves disease. This was making her unable to sleep and prone to a racing heart and anxiety attacks. This isn't good for a hostess who has to cope with two babies, so I really sympathized. We spent a couple of days staying out of her hair as much as we could, going to some of the Smithsonian Institution museums.

To top it off, our hostess was scheduled to take a dose of radioactive iodine. We had to find another place to stay for the last couple of nights because our hostess was literally about to become radioactive, and her instructions advised her to avoid people, and especially children, as much as possible! So we had to scramble a bit.

Another complication -- I deposited a check, which was going to cover some of the expenses for the latter part of our vacation, the Friday before we left, thinking that it would clear in a few days. A week later, though, acting on a hunch, I asked Grace to call the bank, and she found out that the check had not cleared, and in fact the funds would not be available until the tenth calendar day after making the deposit.

We had just set up these new accounts because we were unhappy with our old bank, which was acquired, unilaterally cancelled our overdraft protection arrangement, instituted ridiculously punitive fees, and put in place various floats and back-dating of checks apparently designed specifically to absolutely maximize said fees.

Granted, the ten calendar days, during the holidays, was only five banking business days, but in the world of Check 21 and electronic check clearing it seems just insane to me that an institution would hold a deposit that long. We were able to convince them to make enough of the funds available to cover several check card transactions we had just made, but our old bank -- the one I'm planning to leave -- clears all deposits overnight. It seems that this was technically legal under the Expedited Funds Availability Act, but I am distinctly unhappy with this, and contemplating whether or not I want to close these new accounts immediately and find another bank. (Do any of them not suck?)

Fortunately, I had set aside a little bit of extra money in a savings account at our old bank. Using my other cousin's iPhone, was able to log in to the bank and move that money into checking. Did I mention the iPhone is very cool?

We were able to stay with family friends for the last couple of days in Richmond, Virginia. Along the way I got to make Christmas better for two of their three sons. They had both gotten iPod Shuffles for Christmas. (I wonder how many people got iPod Shuffles for Christmas? It must be in the millions!) iTunes would not run correctly on their Windows XP box. They had taken it to a local Staples and the guru there had spent a whole afternoon trying to get it to work, and failed. The symptoms were this: iTunes apparently installed successfully, but would crash immediately after launching, with the usual "tell Microsoft about the problem" message. QuickTime player would also crash upon launch, with one of several error messages, including a security warning about stack overflow in a Visual C++ library. The uninstall process for iTunes and QuickTime always failed.

Despite a distinct lack of expertise with Windows system administration, I decided to take a crack at it. I had to putz around for a long time. I Googled myself into a frenzy looking for notes from people with similar problems. I downloaded and ran several virus and trojan detectors. I installed every recommended Windows XP update I could find, and cleaned out a bunch of unused software. Nothing seemed to help. Finally, I forcibly uninstalled iTunes and QuickTime (removing everything Apple-related from the registry, and deleting the program directories). I then downloaded a whole series of earlier versions of iTunes, starting with 6.10. This one installed and ran without a hitch, so I started going version-by-version. At some point, I found an installer which said that it could not run properly because the VBScript service was not enabled. This led me to an Apple support article on enabling VBScript, which had probably been turned off by a security product. After that all the installers seemed to work and I was able to get the latest iTunes installed.

The problems seems to be that one of the more recent iTunes installers silently fails if VBScript is not enabled. It runs and seems to believe that it has succeeded, but leaves behind an unusable QuickTime configuration. iTunes needs QuickTime and thus crashes on startup. Somewhere along the way Apple's installers lost the ability to verify that the necessary VBScript service is available.

After finally getting iTunes working, I thought it was all going to come to nothing, because the iPod Shuffle itself was not working. iTunes could see it, and fill it up with music, but when it came time to turn it on and push play, it would just flash a series of alternating green and orange lights and do nothing. A complete reinstall of the Shuffle's firmware didn't help. Running Apple's separately available iPod Shuffle utility designed to fix this problem didn't fix it. I thought we might have to just send the iPod back. But then apparently just toggling the little switch between continuous play and shuffle made it suddenly work. This does not fill me with confidence about the device's firmware, but it was working.

Anyway, I spent a ridiculous five or six hours messing with this, but got a number of hugs in return when it finally worked. I can attribute my success only to being tenacious. Age and tenacity beats the 18-year-old Staples employee FTW!

We also got the opportunity to meet up with my friend Antonio, and had a great chat with him.

Although we gave ourselves what I thought was a sufficient safety margin, planning to arrive at the train station an hour and 45 minutes prior to scheduled departure, we had a couple of delays. We got slightly lost getting back to the train station, and so arrived only 20 minutes before the train was scheduled to arrive. While we were waiting at the light to turn into the station parking lot, it arrived. Then, under two minutes later, before we could even get inside the building, it left. Without us.

So we had another night in Gaithersburg, at a Holiday Inn. That wasn't so bad. It meant I got to watch Iron Chef and soak in the tub. We had to get a hotel shuttle to the nearest metro station, then carry the car seats on the metro, while a friend of my cousin drove our luggage to the train station. He was a huge help.

We spent New Years' Eve on the train. No one got much sleep, and we got back to Toledo on time (about 5 in the morning on the 1st). Then we had a drive back up to Ann Arbor. The drive turned into blizzard conditions. We had to crawl along moving at times just 25 mph, nervously looking at quite a few cars that had slid of the road. But we all made it back safely.

Oh, we did nearly lose one of the babies. At the end of of our train, at the last door, where you could stand and watch the tracks retreating, Grace and I had come back and stood there and looked out the window. "I wonder if that door would open if I pushed the button?" she asked. "No way," I said. "I'm sure there is a security interlock of some kind so if it isn't connected to another car, it won't open. That would be a huge liability issue if there wasn't."

Well, Veronica proved me wrong. We were walking up and down the train and she pushed the button to open the door. Fortunately, I was holding her hand. There is a kind of cage to prevent someone falling out, but it was really just a couple of metal bars and they were spaced far too widely to keep a child from pitching right out the door onto the tracks, from the upper level of a speeding train.

Grace mentioned to the conductor that the rear door was unlocked. She saw, as she described it to me, "a black man turn white." Yes, it was supposed to be locked. He ran back to lock it.

So, that was our Christmas vacation. No casualties but my sanity. Now I just need a vacation from my vacation!

Brussels Sprouts with Toasted Cashews

I made up this recipe after being inspired by an episode of Iron Chef I happened to see on vacation. That was a venison battle, but one of the dishes involved brussels sprouts. My family tells me it is one of the tastiest dishes I've ever concocted. I served it on New Year's Day. It seemed like a very new-year-ish dish.

If you don't like brussels sprouts, it could be because if you cook them by steaming or boiling, they tend to turn into nasty, bitter, sulfurous little cabbages. Cooking them in a dry method with high heat caramelizes them and releases wonderful complex nutty flavors and aromas.

A warning: if you are not used to eating a lot of cruciferous vegetables, this dish may be a bit challenging to digest. You may need to sleep with the windows open!

Serves 8, or me and three other people : )

I served this with mixed greens (collards, kale, and mustard greens) cooked with sage-flavored bulk pork sausage and a loaf of challah with unsweetened butter. We had a Shiraz with it, but that was all wrong; it would have gone much better with a very cold Charonnay or Riesling.

For the spices, before you wind up buying something, see if you've got something useful on hand. I actually used a leftover pumpkin pie spice mix that my wife made for Thanksgiving. I tend to cook by sniffing and tasting the spices and the raw food and deciding what seems like it would go well together. Trust your instincts!

You will need:

• 3 or 4 lbs. fresh brussels sprouts
• 1 cup raw cashews
• Hard cheese (parmesan reggiano, aged gouda, etc) -- enough to yield 1/2 cup grated
• 3 Tbs grapeseed oil
• 2 tsp allspice or pumpkin pie spice mix (allspice, nutmeg, cinammon, cloves)
• 1 tsp red pepper flakes (optional)
• Salt to taste
• A heatproof bowl
• A large frying pan

Grate the cheese. The sprouts should be dry, so if you washed them, dry them completely with paper towels. Cut off the stem ends and make them into 1/4" slices. (Don't worry if some of them fall apart and you have loose bits -- the goal is to give the vegetables a lot of surface area).

Heat half the oil to medium heat and throw in the cashews. Fry until nicely browned, turning constantly to avoid burning (perhaps 3 minutes). Scoop the cashews out into the heatproof bowl (make sure you get all the pieces of nut, or they will burn in the pan and ruin the flavor). Leave some of the nut-flavored oil in the pan. Add the cheese to the hot cashews and stir quickly. Set the nut mixture aside.

Add the remaining oil to the pan and increase the heat to high. Before the oil starts to burn, toss in the brussels sprouts. You want to cook them quickly, turning several times to lightly brown the cut surfaces, until they are softened slightly and a bit brightened in color. They should release a nutty aroma. You don't want them to start cooking down and releasing a lot of liquid. On my stove this took 3 minutes or so, but your stove may vary. Add the allspice and red pepper flakes about halfway through cooking. Add the cashew/cheese mixture and mix. Add salt to taste (I threw in only a small amount, perhaps a quarter-teaspoon). Serve immediately.

If you try this recipe, I'd be interested to know what you think!

Security Warnings with Ubuntu 7.10 (Gutsy) Updates

The most recent updates to Ubuntu suddenly started generating warning messages saying that the patches could not be authenticated.

This is apparently a known bug in package bookkeeping. The procedure for fixing it seemed to be:

1. sudo mv /etc/apt/sources.list /etc/apt/sources.list.backup

2. sudo touch /etc/apt/sources.list

3. Run the package manager and check for updates. You should see none; quit the package manager.

4. sudo rm /etc/apt/sources.list

5. sudo mv /etc/apt/sources.list.backup /etc/apt/sources.list

6. Run the package manager again; this time you should be able to install the updates without warnings.

Weird!

Newegg Makes Good

Newegg is refunding me the camera and case and I was able to find the camera elsewhere in time, so it is all working out OK. They have excellent customer service, so I will no doubt be buying from them again in the future.

Today is the shortest day of the year. It takes a lot of caffeine and St. John's Wort to keep me functioning this time of year. I think the barista at the local Caribou Coffee realized I needed a triple espresso and so I got a free upgrade. Either that or she was thinking "you know, we don't see you in here often enough... you're just not quite as addicted as we'd like you to be!"

This year really was a challenge -- illness, death, and mayhem all around us. Let's hope 2008 is better!

Newegg Screws Up

So, I decided rather late in the season to get my son an inexpensive digital camera for Christmas. I ordered it from Newgg, a company I've had great service from in the past. I bought a motherobard and CPU from them a few years ago; I've ordered various little things like flash drives and hard drives.

They took the order for a camera, case, memory cards, and card reader. I paid with PayPal; everything went through normally. The package went out; it is supposed to be delivered today. I wanted to make sure of that, since we're obviously getting pretty close to Christmas, and my family is also planning on going out of town. In fact, we're taking the train out of town to see family. The plan was for my son to enjoy taking pictures of the trip. So it has to be here, or there wasn't much point.

Today, the day that my UPS package I've been tracking is scheduled for delivery, I got a note saying that it was all just a joke... the package I've been tracking doesn't actually have a camera in it, and I should get a refund in two or three days. Oh, it does have a cheap little camera case in it, which I threw in just because they were shipping me the camera anyway. Which means I just paid $5.24 to ship a $10 camera case.

I'm scrambling to try find the camera locally. I'm fortunate in that my finances aren't that tight right now, but if they were, I'd be completely screwed, because i wouldn't be able to charge the replacement camera on my debit card until the PayPal refund went through.

I can't recall any company I've ever ordered from screwing up an order in quite this way. I mean, I've had items cancelled on me at the last minute, or delayed, but I've never, ever been charged for an item that was not actually shipped. I'm a bit boggled by just what kind of a failure in their IT infrastructure allowed the charge to go through. Wow!

The SOE

I have been looking at Paul Hudak's book The Haskell School of Expression. My first barrier was getting the sample code to work under Ubuntu "Gutsy Gibbon." While I highly recommend Graham Hutton's book, which I think of as the K&R of Haskell, SOE is more of a tutorial and may be more useful to people who learn by doing. I got a bit of assistance from Paul Liu. It turns out that in addition to installing ghc using sudo apt-get install ghc, I needed to install libghc6-opengl-dev.

Without this, when I tried to configure GLFW using runhaskell Setup.hs configure, I got an error that said Setup.hs: cannot satisfy dependency OpenGL>=2.1. I went down a dead end of trying to figure out what kind of package I needed to install to give me the right OpenGL libraries, but the key was realizing that it is looking for the Haskell OpenGL library, not the system library.

This is not specifically mentioned in any of the docs I found online -- the only place I found it mentioned was on the Debian package list -- so I am mentioning it here! Maybe if someone with the same issue is Googling for the answer, they will now find it quicker.

Getting Back Online with Music Theory

Well, this hellacious year is almost over. Most of my spare time has been spent on the archiving and family history project, which I'm documenting at The Marcella Armstrong Memorial Collection. That project is bearing fruit in that I have almost a thousand images scanned, and so I'll be giving some of the originals to family members. I've created some beautiful prints, as well, and over Christmas I'm hoping to share some of the images with family members in the form of slideshows.

Meanwhile, in the back of my mind I'm thinking about how to get back into Haskell programming. Music theory keeps coming to mind.

The problem with music theory is that it is really a collection of ad hoc convenient rules and relationships in the guise of a coherent theory. The various named entities have completely inconsistent nomenclatures. You kind of get used to this when you learn to play chords and transcribe songs, but it is easy to forget how inconsistent it is and thus confusing for students.

For example, intervals. A major second is an interval also known as a whole step. If you start on the root note of a scale, say, a C major scale, the C is known as the root or "first." If you add a major second to a first, you might imagine that you'd get to a third note of the scale. But, no, you have a second note.

Does that mean the steps in the scale are off by one in their naming (that is, the first is really the zeroth?) No, because if you add two major seconds to the root, you have a third, not a fourth. It is the naming of the intervals that are off.

So can you just start the intervals at zero instead of one? Well, kind of. A minor second is a half-step, or the difference between adjacent keys on a piano. Call that 0.5. A major second is then 1.0. But this still doesn't explain intervals, because a "fourth" is actually three whole steps and a half step. And if you add a third and a third, you don't have a sixth, you have a flat seventh!

So it keeps coming back to me that I should try to codify the rules of intervals and chords in some bits of Haskell code. It might even be of use to geeks trying to understand music theory.

Don't even get me started on time signatures or tuning!

Memento Mori

So, I have been falling behind on all sorts of commitments -- I was excited by the prospect of working through a bunch of Haskell exercises and get more serious about learning Haskell; I was learning all kinds of new jazz chord voicings in my guitar lessons; I had plans to record another podcast with original music.

Instead life intervened, and my mother died after a brief, unexpected illness, and then Grace's father died just under two weeks later. We've had a lot of death recently -- my grandmother died two years ago, at the age of 102, and Grace's brother died at the age of 40 just last year.

"Memento Mori" means roughly "remember, you will die." There's nothing quite like sitting in a room with the dead body of your mother to make this clear. It has been my task this year to think pretty hard about this, and to try to start planning for it, beginning what I hope will be my "ars moriendi" -- the art of dying well. And, I hope, a lot later.

I've been living on a kind of "split screen" for the last few months -- on the one hand, working on retirement plans and investments for my children's education and imagining what Grace and I are going to do for the next five, ten, twenty, or forty years, and on the other hand preparing our wills and making sure we are properly insured. It's been a strange combination of unnerving and reassuring. My grandmother made it to 102, and I have her genes, so I could have 60 years or more to live. Or I could take after my mother, and have 30.

Or none.

On the way to work this morning, as often happens, the light changed for me to make a left turn. I did my usual deep breath, look both ways, count slowly to five -- to wait for whoever was going to blast through the red light to go ahead and do so -- and then started to enter the intersection. At the ten second mark a woman in an SUV blasted through the red light, going way over the speed limit, talking on her cell phone. Missing me by only a few feet.

Memento, mori.

A year ago someone did this and I was hit by such an attack of road rage I actually chased him down, cut him off and forced him over to the side of the road, then got out of the car and chewed him out for nearly leaving my children fatherless. I can't advocate that behavior. Today I took a deep breath and let it go.

It would be a stupid way to die. But so is cancer, or heart disease. And we don't have control over everything. And believing that we do is a recipe for a heart attack.

There is good news in our lives too -- we just celebrated our sixth wedding anniversary and baby Sam's first birthday and baby Veronica's third birthday -- but these celebrations have all been kind of subdued.

There have been a whole bunch of miscellaneous estate issues -- fortunately my stepfather and the estate attorney have been managing most of this -- but my stepfather wants to sell the house he and my mother shared in the very short term, and so is trying to dispose of my mother's personal effects very quickly.

This means a house full of furniture, clothes, and personal effects. Since I'm the son that lives a mere 250 miles away, instead of 2,000, I'm the one that has to figure out how to triage everything and move anything we want to save into our rather cramped and cluttered apartment. Which means a major purge of our existing clutter, and also coming to terms with letting go of almost all of my mother's personal effects.

Along the way I discovered that my mother and my grandmother had amassed a huge collection of family photos and documents, going back several generations. In addition, hundreds of documents: letters, journals, autobiographies, even short stories.

We have pictures of people I think are my son's great, great, great, great grandparents. I have not identified everyone yet, but there may even be pictures of a five-greats grandparent. My grandmother was a member of the Daughters of the American Revolution, which means she can trace her ancestry back to 1776. Which means my daughter can, too. That's good, because it appears from the organization's somewhat controversial history that they could use more black members!

There are Civil-war era photos. Cyanotypes from around 1900. Thousands of photographs -- perhaps 10,000. The oldest ones are mostly in pretty good shape, but many of the color photos, for example instant photographs from the 1970s, are fading badly. And there are some serious preservation issues -- photos that were recently annotated in ballpoint pen ink, which is acidic and eats through the paper until it stains the emulsion. Photos torn from albums and scotch-taped into new albums, or bundled together with paper clips or rubber bands and stuffed into acidic paper envelopes and shoe boxes.

I decided, and Grace concurred, that I was going to engage on a preservation and archiving project. We can't let the collection of family history end with my generation. So I have embarked on that project, which will consist of organizing, cataloging and propagating both the original artifacts and digital derived works.

So, for the immediate future, this project is now my highest priority. My other projects, blogs, and commitments are largely at a standstill. I apologize to everyone who I've ignored or failed to follow up with on some promise or another. But I think my children and grandchildren will approve.

Anyone interested can follow my progress on my blog, The Marcella Armstrong Memorial Collection.

Another Death in the Family

We just received word that Grace's father, my father-in law, died today at about 6:10 p.m. He died just 13 days after my mother.

This means that this month we lost both my mother and my wife's father. Our children just lost two of their four grandparents in a span of just under two weeks. Isaac, at 13, was old enough to get to know them a little bit, but Veronica and Sam, at 2 years 10 months and 10 months, respectively, will never really get to know them.

It's been a hell of a month. Since bad things usually come in threes, it makes me wonder what we're facing next! Maybe it's my turn?

Eulogy for My Mother

(This is not about Haskell or even programming per se, but it is, in part, about the interaction between technology and human lives and the limits of that technology, and about focusing on what is important in life. So I thought the Haskell community might forgive me for posting it in this forum. I hope to get back to writing about Haskell soon).

Dear Mom,

In the past few years, whenever something happened in my life, good or bad, I always got the urge to call you and bring you up-to-date, reassure you, and get your reassurance and advice. For the past few weeks, I've had that urge to call you constantly, because a lot of big changes have been happening. Now I can't call you, but I can still write you a letter. I can't mail it to you, but here is what it says.

The big news is that you have died. This was quite a shock to everyone. It happened very fast. You had a lot of things going on -- oxygen masks, IVs, tests, pain medication, sedatives, all kinds of doctors and nurses, and lots of family members with worried expressions on their faces. It must have been very confusing, especially in the last few days, so I thought I would explain to you what happened in case you were confused.

About two years ago you were treated for breast cancer with a combination of surgery, chemotherapy and radiation. I know that was very hard on you, but the treatment seemed to be quite successful. You recovered some energy and were able to spend quite a bit of time in the last two years traveling with your husband. You were getting regular follow-up care to make sure the cancer did not come back.

About three weeks ago, after a short trip to Chataqua, New York, you were exercising with your friends at the YMCA. Things didn't feel right, though; you had been complaining about a feeling of bloating and pain in your abdomen. You said that you must have eaten something that didn't agree with you. You were only a few days away from your regular follow-up visit with your doctor. But when you called and told him about the problem, he suggested you go to the Emergency Room immediately.

From that point everything started moving with frightening speed. You had a CAT scan, and it showed tumors in your abdomen. It isn't exactly clear whether these might have been caused by cancer cells from the breast cancer or whether it was another, separate case of ovarian cancer. It doesn't really matter much at this point.

It would be easy to get very angry at your doctors and blame them. I find it a bit hard to believe that oncologists screening someone regularly for cancer would fail to detect such an advanced case of cancer. It isn't like you came down with a rare tropical disease that was outside their specialty. If anything I would have expected them to suspect cancer and do extra tests to rule it out even when it wasn't there. However, it is important to note that this kind of cancer, in the abdominal cavity, often has no detectable symptoms at all until it is very advanced. And I remind myself that the treatment you got two years ago did restore your health and give you two good and enjoyable years you would not have had otherwise.

Anyway, the next thing was that you went down to the Magee-Womens hospital in Pittsburgh for some tests. The plan then was to find out what was going on and make a treatment plan. But your pain got rapidly worse. You were admitted. I spoke to you on the phone each day for a couple of days. I was making plans to come and trying to arrange with my father and brother to come too. We got the word that you had been moved to the intensive care unit, and so everything went into high gear. We moved up our visit. My brother and my father arranged to fly out. We all got into Pittsburgh the evening of Thursday the 9th of August, just after a series of severe storms had produced tornadoes and flooding. When my father and brother got in by plane around midnight they immediately went to the hospital. Grace and the children and I all came to visit too, in the middle of the night.

You were miserable and frightened in the ICU. You were curled up on your side, wearing an oxygen mask, with your eyes tightly closed. You were able to talk, but mostly we just wanted to hold your hand. It took a while for us to figure out exactly what was going on. You had a a partially collapsed lung, and fluid building up around your lung. It was hard for you to breathe. You had pneumonia in one lung. You also had some sort of kidney infection. But with tubes and antibiotics and a lot of moral support over the next day or so you improved. You got your eyes opened and we were able to sit and talk with you. We talked about how you were doing, what we knew and didn't know about your condition. We told you about our families and how well everyone was doing. We talked about dying and what we thought it might be like, and what you believed and we believed would happen to you after you died. We told you we loved you.

You were moved back into a regular hospital room. This was still not a very comfortable environment but it was a far less frightening place than the ICU. Over the next couple of days you had a lot of visitors -- your husband Dick, your daughter-in-law Carolyn, my father Richard, my brother, your niece Linda, your nephew David, and Grace and the kids. We arranged for people to stay with you round-the-clock. Often one of us would be with you in your room while another one of us caught a quick nap in the waiting room next door.

One night while my brother was sitting with you during the middle of the night you began fighting to get out of bed. My brother tried to use the call button to get help but it didn't work. He had to run out into the hall and yell for help. You managed to get partly out of bed and it is a miracle you didn't tear out an IV. But you had a breathing crisis and you had to go back onto the high-pressure oxygen mask. You hated that mask because it was painful; it rubbed your face raw and dried out your lips and mouth terribly. For a few days it went back and forth like that. You'd improve a little bit; you got a transfusion and some new drugs and that seemed to help. But you'd get worse in the middle of the night.

The CAT scan and biopsy was postponed again and again. The doctors found that you had blood clots in your leg and also that something had gone wrong with your heart. It was very weak and it appears that you may have had a heart attack, or maybe it was damage from your previous chemo or radiation. The radiation in particular may have been poorly administered. Or maybe it was all three. They put you on heparin to thin your blood. They decided they could not do a biopsy. At one point the doctors were considering putting you back into the ICU but you did not want to go back. What you wanted was to leave the hospital, and to go up to a nursing home in Erie so that you could die in comfortable surroundings.

This brings me to the point of talking about your wishes. You made clear both in your written directives that if there was no chance of recovery, you did not want invasive procedures or extreme measures taken that would just serve to prolong your life. You told these things to the doctors as well -- no tubes for breathing or feeding, and no zapping your heart with electricity if it stopped. It is one thing to check some boxes on a form, but I think it is quite another thing to realize that it is time for these directives to be carried out. I was, in fact, awed by your bravery in sticking to your principles.

So, we next focused all our attention on trying to get you moved to Erie. I have to confess that I was terrified that you would die before we could carry out your wishes. But there was only one day's delay, and on the morning of Tuesday, August 14th, the ambulance crew came to take you to Erie. I rode in the front of the ambulance with the driver while you were in the back with the nurse. For me it was the longest chunk of uninterrupted quiet time I had gotten since leaving for Pittsburgh and I finally broke down crying for a while.

I thought that the stress on the family was going to let up and that you would be in good care. But apparently hospice care in Michigan and Nevada and California is much different than the hospice care arrangement we had in Erie. They got you settled comfortably into a room at the Manchester Presbyterian Lodge. The social worker and hospice nurses and nursing home administrator and head nurse greeted us. But things did not go quite like we planned.

All of your IVs and needles and big heavy oxygen masks were removed. Instead you just had a simple cannula in your nose for oxygen. Your pain medication was changed to liquid morphine by mouth. But after 24 hours it became clear that things weren't quite right. You went from being able to communicate to sleeping around the clock. You were over-medicated. It also became clear that without some orders from the doctor, guidance from the hospice organization, or specific requests from family, the nursing staff at the home was not going to give you special care. For example, they still had you on a regular diet. They would plunk down a meatball sub on your table and take it away an hour later. Because you were so heavily drugged they would not even try to get fluids into you. You asked for pen and paper to write some notes but you were so sedated that we could only read a few words of what you wrote. You wanted to communicate but you couldn't. The system was failing you again.

So once again the family went into high gear. We started a round-the-clock visiting schedule with you. We had to feed you because the nurses were not trying hard enough. We got your diet changed to food that you were better able to eat. We started trying to get your doctor to change your prescription, but we were unable to get the doctor out to examine you in person and change your orders until the evening of your third day there. Your condition was changing for the worse so rapidly that we found this completely unacceptable. I began refusing part of your medication on your behalf.

This was probably the most nerve-wracking experience of my life, because above all I did not want you to be in any pain, but I thought that you deserved to share your last days with your family and friends. I think it was the right decision, though. You became alert again, while reporting that you were not in any pain. You were able to visit with family and friends. We got round-the-clock visitation going again. Carolyn came back, and brought her daughter Alicia. You had live music. You had grandchildren visit. You had a couple of good days.

We still felt like we had to watch over everything. Even after your doctor came back and changed your medication orders, although my father followed him down to the nurses station to see what he wrote, what he wrote was not what he had just agreed to write. Your doctor was literally attempting to euthanize you -- not to help you live out your last days as well as possible, but to drug you up and ignore you until you died.

I felt like we were having to act as doctor and nurse and social worker ourselves, all the time. Between the lack of sleep and the nervousness, I completely exhausted myself, to the point where I could not walk and began having hallucinations from lack of sleep. I developed an ulcer. The rest of us were also exhausted. It was the single most stressful week of my life to date.

You gradually became weaker. Your heart rate went up and your blood pressure went down. You began to accumulate more fluids in your lungs. We got your medication raised again so that there was no possibility you would be in any pain. The decision was made that you could not receive anything more by mouth to eat or drink. On Saturday evening at about 6:30 p.m. you died. It was as peaceful and graceful a death as we could make it. I was not there, but your husband and his daughter Alicia were there. Alicia was playing music.

I just want to say a couple more things.

First, that the way you faced your death was an inspiration to me and to everyone around you.

Second, when my father and I were visiting you in the ICU, you said that you were very proud that you had always had the ability to forgive everyone who had harmed you. I would like to ask you one last favor. I want you to forgive the doctor in Erie who displayed such indifference to your condition. I want you to forgive the other doctors that failed to detect your cancer. I even want you to forgive the doctor in Pittsburgh who had such terrible bedside manner that he burst into your hospital room, explained loudly that your heart was failing and that there was nothing more they could do, and left. I will forgive all these people. It might take me a while longer, though. I'm not quite as good at it as you were.

Third, I'd like you to remember that we had our share of doctors with poor bedside manner and nurses who treated you like an annoyance, there were also quite a few people who were very kind to you. The hospice nurse who was with you on your last day was wonderful. The palliative care physicians in Pittsburgh were wonderful. They spoke very calmly to you and asked you if you had any unfinished business. You said that you wanted everyone to know that you loved them.

And last, I just want to thank you. You were a single mother in a difficult time. You had a hard and complicated life. You raised me and my brother. You did a great job. You re-married and brought new fathers into our lives. You cared for my stepfather Wence, and you cared for your mother, and you cared for your husband Dick. You enriched the lives of everyone around you. You had a lot of friends and you were well-loved, and you still are.

We'll go on. We'll miss you terribly. I wish you had gotten more time to enjoy your grandchildren. I wish we had gotten more time to spend with you. You said that you were concerned about the state of the world. The world will go on. It will be OK. We'll be OK. And we know that you are OK now too.

I'll see you again, before too long. And I'll be in touch.

Love,

Paul

A Hard Dive Headache

This isn't exactly programming-related per se, but I present henceforth my story demonstrating how hardware compatibility issues trump software compatibility any day!

I have a PC and a Mac on my desk at home.

I have a small handful of spare hard drives I use to back up these computers plus two laptops.

I try to buy hard drives when they hit a price bang-for-buck "sweet spot" -- this year, that is in the 250G range; a couple of years ago, it was in the 40G range. So I have a couple of 250G La Cie enclosures. I also have a couple of fanless "Mad Dog Multimedia" enclosures which I bought empty and filled with Seagate 40G drives. They're both USB/FireWire, which is ideal for moving things around between Macs and PCs.

All of them come with external power bricks. The power bricks for all four drives are nearly identical, physically, black boxes with removable AC cables and attached DC cables. The boxes have little green LEDs. They all have identical DIN-4 round 4-pin connectors. These have a little notch on one side and a flat spot on the other, to make sure you orient them correctly when you plug them in, and identical DIN-4 connectors that plug into the drives.

They're all plugged in to power strips on my desk.

The "Sunpro Powmax" bricks that came with the Mad Dog enclosures have a little diagram on them that looks like this (sorry, I don't have a digital camera that will zoom in close, or I'd take a picture).

            _____ 
          /       \
GND ---- | *     * | ---- GND
+12V --- | *     * | ---- +5V
          \       /
           --__--

On the diagram above, the flat side of the connector is shown at the top, and the little "notch" is not shown, but it goes at the bottom.

The LaCie bricks, on the other hand, made by "Sunfone" instead of "Sunpro," have a little diagram that look like this:

     --  ---
    /  \/   \
2  | *     * | 1  PIN 1:5VDC
4  | *     * | 3  PIN 2:12VDC
    \_______/     PIN 3:GND
                  PIN 4:GND

Where the flat part of the circle is at the bottom and the little notch is at the top, although this is actually pretty hard to distinguish in the diagram.

I'll flip them around for you so the orientation of the circular connectors is the same, and label the pins more clearly:

    __-----__           __-----__
  /           \       /           \
 |   G     G   |     |    G    G   | 
 |  +12V  +5V  |     |   +5V  +12V |
  \           /       \           /
    -___/\__-           -___/\__-

Just let that sink in for a moment.

It took me a while to guess why my hard drive, a 250G La Cie drive, wouldn't mount. Finally I started looking at the power bricks. You know that feeling you get when you find out that you've just broken something expensive?

I guess it could have been worse; it could have fried the PC's motherboard too, via the USB port. Who designs this crap? (I know who builds it, but I can't very well blame this on the manufacturers in China; it appears they both did a wonderful job manufacturing exactly what was specified).

Both power supplies have all the appropriate regulatory agency approvals: CE, FCC, UL, and some others I don't recognize. But, of course, they don't actually conform to any particular wiring standard.

After determining that the La Cie drive was out of warranty, and given that the warranty was unlikely to cover this kind of idiocy, I decided to try opening up the enclosure in the hopes that it might have some kind of fuse that I could replace, and that the drive itself was OK.

Surely they must have some kind of fuse in the enclosure's circuitry, right?

I'll bet you a new hard drive that they don't.

Well, the La Cie enclosure is a pretty nice enclosure, but it isn't designed to come apart. They have part of the drive covered with an easily torn (yes, I should know) adhesive foil, and mounted on some kind of sticky silicone anti-vibration pad. I could try stuffing a new drive in it at some point.

The dead drive smells like burning plastic. There's one little surface-mount component on the back that is visibly burnt. If this were 1980, with my knowledge I gained putting together Radio Shack 250-in-one electronic experiment kits, I would be able to tell you if it is a resistor or a capacitor or a diode. If this were 1980, I could probably even solder on a new part. But it is 2007 and all the little black parts look the same, and even if I could find a replacement component I doubt I could solder it on.

Needless to say, the drive would not spin up when I put it in the "Mad Dog" enclosure.

So I put the 40G drive back into the "Mad Dog" enclosure, and powered it up.

I'll give you two guesses which kind of power brick I plugged it into.

My office smells like burning surface-mount components. There's a nearly identical little burnt component on the other drive. It's in a slightly different place on the 40G drive.

I had some files on those drives that I wanted. The 250G drive had mostly unimportant stuff, but a few gigabytes of important stuff.

I'm no fool. I kept a backups of that handful of important files.

On another hard drive. Guess which one?

I've got a headache.

I can't in good conscience ask Seagate to fix them under warranty. Can this kind of thing even be repaired any more? It seems like it should be possible. But I'll have to look into that later.

I'm going to bed. Ever feel like you want to yell at someone for being an idiot, but the only one you can find is between your own chair and keyboard?

A Wonderful Waste of Time

I just encountered this great toy to challenge your three-dimensional geometry skills. The game in effect asks you to pretend you're creating an Escher-like puzzle, producing a an object that only looks right from three precise vantage points. Ever wanted to pretend to be a 3D rendering optimizer function? Now's your chance!

Warning: if you start it, you won't be able to stop until you complete all the puzzles. Which I did, although it took me an embarassingly long time.

Haskell for the Short Attention Span: A Simple File Filter

Well, there is more to say about run-length encoding and other interesting examples I was sent in response to my last article, but today I've been working on another little real-world problem. I've been able to spend a little bit of time on #haskell and the gang there has been extremely helpful. The task: filter a binary log file, turning it into a text representation, and do some stateful validation. I'm not going to show you all the code (it is pretty boring), but here are the minor pitfalls and helpful suggestions I encountered along the way. First, I started with a very basic file filter example from the Haskell 98 report:

main = do
    putStr "Input file: "
    ifile <- getLine 
    putStr "Output file: "
    ofile <- getLine 
    s <- readFile ifile 
    writeFile ofile (filter isAscii s)
    putStr "Filtering successful\n"

Pretty simple. As a sanity check, I wanted to run this example on a data file. The first thing I found is that isAscii is not in the default namespace. To use this program in GHC you'll need to import the module Char. You can learn this via Hoogle. I've started using Hoogle quite a bit; it is a great little tool! You can find Hoogle here. You can also get to it directly through #haskell via LambdaBot. Or you can put it right in your GHCi. Lambdabot lives here.

Anyway, the next thing I found was that, at least under Cygwin, unless you explicitly open a file in binary mode, you might encounter problems. One I should have anticipated -- you might see line feed translation. But there are apparently others -- like silent truncation of the data! Somewhere in the guts of Cygwin, or maybe Windows, control codes designed back in the Jurassic days of computing are still being honored. Input was terminating on (apparently) an EOF character in my binary file. Another suggestion from one of the very smart folks on #haskell.

I'll just use readBinaryFile instead of readFile, right? Well, no, readBinaryFile is part of MissingH, a third-party library. Back to #haskell, where I was advised that I could roll my own. Importing System.IO, I use the following snippet:

readBinaryFile s = System.IO.openBinaryFile s
    System.IO.ReadMode >>= System.IO.hGetContents

This works fine, although I would humbly suggest that GHC's standard library should provide easy access to binary files; while the solution is pretty trivial, it is a wart.

I next stubbed my toe on printf. It is available in module Text.Printf and provides a type-safe version of C's printf. As a long-time C and C++ programmer, you'd think I'd know just about everything there is to know about printf. So, I rapidly gave it a format string "%02X" and passed it a char. Apparently the uppercase X to produce a hex representation with uppercase A-F is not supported (grrr). Another minor wart -- if you provide printf, it should behave like printf -- but we'll move on.

Per more chatting on #haskell I was given this one liner to dump binary data in a nicely formatted way:

writeFile ofile (concat $ zipWith (printf "%02x %s") s (cycle $ replicate 19 "" ++ ["\n"]))

I want to take a moment to talk about how it works. First, cycle $ replicate 19 "" ++ ["\n"]. The replicate function gives us a list of 19 empty strings, which we then concatenate with a newline. Applying cycle to this list treats it as an infinitely repeating circular list of strings, where every twentieth is a newline. These arguments are then fed to printf using zipWith. zipWith is an interesting function: while zip takes two lists and generates a list of pairs produced by assembling the list elements into tuples, zipWith doesn't tupleize the elements; instead it feeds the elements to the provided function, and makes a list of the results.

While this worked, it was interesting enough that I wanted to play with it using GHCi. But I had to give up on that; I kept tripping over the type checker. While I appreciate the masochistic joys of programming and the safety that comes with it, it can be frustrating for programmers with experience in, say, Ruby, or even C.

For example:

let xs = [1..100]
let ys = take 100 (cycle $ replicate 19 "" ++ ["\n"])
zipWith (printf "%02x %s") xs ys

GHC replies:

Ambiguous type variable `c' in the constraint:
`PrintfType c' arising from use of `printf' at <interactive>:1:9-24
Probable fix: add a type signature that fixes these type variable(s)

Ugh. Using the :t command in GHC it is easy to see that GHC thinks the type of xs is [Integer] and ys is [[Char]] (a list of list of chars, also known as a list of strings). If I put roughly the same code in a Literate Haskell source file and ask GHC to load it, I get:

Ambiguous type variable `a' in the constraints:
  `Enum a'
    arising from the arithmetic sequence `1 .. 100'
    at E:\toy.lhs:3:5-12
  `Num a' arising from the literal `100' at E:\toy.lhs:3:9-11
  `PrintfArg a' arising from use of `printf' at E:\toy.lhs:5:18-33
Possible cause: the monomorphism restriction applied to the following:
  xs :: [a] (bound at E:\toy.lhs:3:0)
Probable fix: give these definition(s) an explicit type signature
              or use -fno-monomorphism-restriction

followed immediately by:

Ambiguous type variable `c' in the constraint:
  `PrintfType c' arising from use of `printf' at E:\toy.lhs:5:18-33
Possible cause: the monomorphism restriction applied to the following:
  result :: [c] (bound at E:\toy.lhs:5:0)
Probable fix: give these definition(s) an explicit type signature
              or use -fno-monomorphism-restriction

Failed, modules loaded: none.

Wow. I'd say that is not really a newbie-friendly error message. However, this printf works fine in my real program. I'm not certain why, and I'm not going to dive into it too deeply right now. But here's a simpler type checking example: while C is strongly typed, you can treat numbers as chars and vice-versa, as long as you keep integral promotion and sign extension in mind. GHC is a harsher mistress. Let's say we want to pattern-match on our binary data. The value 16 in my binary data is DLE, which stands for Data Link Escape; it is often used in serial data to indicate packet boundaries, while inside the payload, it will be escaped (doubled). So here's a little pattern to remove doubled DLEs:

de_dle (16:16:xs) = ...

Simple enough, right? No, to Haskell a number and a Char are not interchangeable. Back to #haskell, where I got a quick explanation of the type checker's error messages. I turned the numbers into chars:

de_dle ('\16':'\16':xs) = ...

And that works just beautifully.

Anyway, to make a long story shorter, I was able to write my file filter, which does some nice pattern matching and formatted out. The problems I had while developing that were the kind I like: problems choosing my algorithm properly, not problems fighting with the language. The runtime was quite helpful here; while processing the file, if I hit a case at runtime which my patterns did not handle, I got a runtime warning about non-exhaustive patterns. That led me to reorganize my patterns, and the result was much clearer.

There was one more minor pitfall remaining. I compiled my program using GHC, but when I ran it, instead of my prompts for input and output filenames, I got nothing! Haskell was silently waiting for input. Back to #haskell. To make the output show up, I had to import System.IO and do hSetBuffering stdout NoBuffering. (Alternately, I could flush stdout immediately after each putStr, but that seems even uglier). I hope that saves someone a little aggravation.

Speaking of aggravation, how did it all come out? Well, the original binary log file is about six megabytes. Since I was doing this by hand, I was more concerned that the program ran correctly than that it ran fast; I would have been satisfied with anything under a half-hour. In fact, without any attempts at optimization at all, my filter ran in under thirty seconds, which is more than fast enough for an ad hoc little tool. I started out trying to do this task using some regular expressions in vi, and that was quickly going nowhere, and taking forever to do it. The vi that came with Cygwin doesn't support some of the more advanced regular expressions features (there is no {x,y} syntax for specifying the number of repeats of a pattern). Notepad++, my workhorse Windows text editor, also doesn't support this syntax. Without this the regexes were becoming hideous, although in (say) Perl they would have been rather simple. But I was able to use Haskell instead, thanks to GHC and the kind folks on #haskell!

Haskell for the Short Attention Span: Run-Length Encoding, Part 3

This isn't a big improvement, but it occurs to me that my implementation from last time:

>deRLE :: [Integer] -> Bool -> Integer -> [Integer]
>deRLE [] False _ = []
>deRLE points False max = deRLEHelper points max
>deRLE points True max = deRLEHelper (0 : points) max

>deRLEHelper :: [Integer] -> Integer -> [Integer]
>deRLEHelper [] max = []
>deRLEHelper [unmatched] max = [unmatched..max]
>deRLEHelper (start:end:rest) max = [start..end - 1]
>    ++ deRLEHelper rest max

can be yet shorter; since deRLEHelper handles the empty list specially, deRLE doesn't need to:

>deRLE :: [Integer] -> Bool -> Integer -> [Integer]
>deRLE points False max = deRLEHelper points max
>deRLE points True max = deRLEHelper (0 : points) max

That way only the function that actually recurses down to the empty list has to explicitly handle it.

Can we make it even more concise and still retain the self-documenting aspects of the code? I'll have to come back to that question. It's time to respond to comments. When I first mentioned my triplify function, which breaks a string into strings of three characters, I wrote "I'm sure there is a very clever one-line fold that applies take 3 in an mind-expanding manner, but I was unable to find it." I got some great suggestions. The first, due to Cale Gibbard, was:

map (take 3) . takeWhile (not . null) . iterate (drop 3)

Let's look at it piecewise as a pipeline, from right to left.

First, our list/string (for example, "ABCDEFG") is fed to the function iterate along with (drop 3). Iterate is an interesting function. I tried to understand it a few months ago by reading the description at zvon.org, which says that iterate

creates an infinite list where the first item is calculated by applying the function on the second argument, the second item by applying the function on the previous result and so on.

They give the following example:

Input: take 10 (iterate (2*) 1)
Output: [1,2,4,8,16,32,64,128,256,512]

Note that if the behavior followed the description, the output would be [2,4,8,16,32,64,128,256,512,1024]. Their description is wrong, although the output matches GHCi's output. What does iterate really do? I'll describe it in the form of a poem.

iterate takes a function and a value, and returns a list consisting of:

the value

followed by
  the value of applying the function to the value

followed by
  the value of applying the function to
    the value of applying the function to the value

followed by
  the value of applying the function to
    the value of applying the function to
      the value of applying the function to the value
      
it will keep talking
  even when
    it has nothing
      new to say
        so please don't
        ask it any
          open-ended
            questions...

That is, I can't evaluate

iterate (drop 3) "ABCDEFG"

because even though the input is finite, the output isn't; there's no termination to the recursion. However, I can evaluate part of the result:

take 4 (iterate (drop 3) "ABCDEFG")
["ABCDEFG","DEFG","G",""]

And because of Haskell's lazy evaluation, we can feed this infinite result "upstream" to be used in the remainder of our calculation, and it will work fine as long as it doesn't evaluate the complete result. The next stage in our "points-free" pipeline is:

takeWhile (not . null)

The takeWhile function operates on a list, returning a list made up of its values as long as they match the predicate supplied. This has the effect of squashing the overly talkative iterate, because we stop evaluating its results as soon as it returns the first null value. Lazy evaluation is so cool!

Now, our finite list of strings is handed off to map (take 3), which generates a list of 3-character prefixes. Here's the whole process:

iterate (drop 3) "ABCDEFG"

yields an infinite list beginning ["ABCDEFG","DEFG","G",""];

takeWhile (not . null)

applied to the above will take elements from the list until it hits the first null: ["ABCDEFG","DEFG","G"]

map (take 3)

applied to the above will give us ["ABC","DEF","G"]. Note that the short remainder is preserved.

>

That seems so useful, I'm surprised it isn't a function in the Standard Prelude, called groupsOf. Although it would also be nice to have a version which didn't keep the short remainder, perhaps groupsOfOnly.

I got another suggestion from a user called "kirby." He suggested the function

takeWhile (\="") . List . unfoldr (Just . splitAt 3)

Wow, my first unfoldr! I'm going to have to stop there for today, while I learn a little something about Just, Nothing, and the Maybe warm fuzzy thing. My first funster!