Posts for August 2007

2007-08-01: mdfrm-utf8 1.1

Tonight I got my first spam that used RFC 2047 encoding in a character set that Encode couldn't recognize. In retrospect, I'm surprised it took this long. It turns out that Encode throws an exception when this happens, so mdfrm-utf8 now catches that exception and displays the raw encoded header as the best available option.

You can get the latest version from the mdfrm distribution page.

2007-08-02: Long time no write

So, as you may have noticed, I haven't been posting much lately, which is because I've been pretty much swamped.

Mostly, it's work. We've had three people resign from the group I work with at Stanford over the past four months or so, which has left us severely short-staffed given that we were down two bodies before that. I'm also trying hard to finish up a major component of the largest work project I've ever tackled (completing our migration to Kerberos v5 and turning off Kerberos v4), and after lots of testing false starts and lots of tweaks, we're finally on target to switch over our administrative master and the source of all password propagation to other environments to Kerberos v5. As you can imagine, I have a list of things to be finished and tested a mile long.

Then, last week, I finally got a much-needed vacation, during which I was planning on doing lots of things for non-work projects like Debian and during which I actually did almost nothing beyond talk to my friends, write, and read. Which was the right choice, as it always is when that happens, but which doesn't help any in getting caught up.

So, I have a ton of OpenAFS patches to apply, a new release of remctl to get out, a new release of kstart to get out, Debian Policy work that desperately needs attention, a new release of lintian that needs to go out, various non-Kerberos bits of work, five book reviews to write for things I finished on vacation, two book reviews and a magazine review already written that need to be posted, and doubtless other things that I'm forgetting. At least I now have most of that written down so that I can start prioritizing.

If all goes according to plan, that means that the activity increase will turn into a temporary flurry as I get caught up with things. I'm hoping for a very productive month. Of course, I also just got a new digital camera, and another book order just shipped.

Really, about four weekends in a row would be lovely right now. Not vacation, since that's a different-feeling sort of thing with time stretching out in front of one and an obligation to use those precious vacation days to really relax. No, I want the sort of short, quality non-work time that I get on weekends, where I get various bits of non-work work done and catch up on to-do lists. I just want four of them in a row.

2007-08-04: New camera

I finally took the plunge and got a Canon PowerShot SD750, and finally this weekend had a chance to start playing with it. Here are a few notes on using it with Debian.

Gnome (gnome-volume-manager, in particular) just deals with the camera quite well. Connect it to its USB cable, connect the USB cable to the computer, and gnome-volume-manager pops up a dialog box asking if you want to import the pictures. You do need to have gthumb installed or importing does nothing without an error message.

The default high-quality large size is probably overkill. I'm not sure if I'm going to keep the huge images or not. ImageMagick is, as usual, my favorite program to do resizing and similar transformations (specifically the convert command-line utility). The camera writes out the standard Exif image attributes. After looking through the available options for an Exif tag editor, I picked exiv2 since it would let me edit Exif comments. Unfortunately, you have to know the tag names in order to use it (some shortcuts for common options would be nice), but there are useful examples in the man page.

The first application I wanted it for was to get cover images for book reviews when the publisher and similar information databases don't already conveniently have one. That took a bit of experimentation, since I can't ues flash (it reflects too much off the cover) and it's hard to get enough light otherwise for the camera to not have to increase the shutter speed and hence blur the picture if you're holding the camera in your hand. Putting the books on a TV tray on my balcony in the direct sunlight worked out wonderfully, though, and I'm very happy with the results. I resized the pictures to 1024x768, then used ImageMagick's display program to crop them and shrunk them down to the size I use for thumbnails on review pages. For the most part, except where I didn't get the book quite square, you can't tell the difference between my pictures and cover art from the publisher.

I'm probably not going to use the camera very much regularly, as I'm not that much of a camera person, but being able to capture cover images is really cool. And it can go with me the next time I go on vacation. I might try taking some video of waves crashing over rocks at Yachats during my October vacation.

2007-08-07: krb5-sync 0.7

It turns out that if you pass an unqualified principal without an instance into the kname_parse function of MIT Kerberos's libkrb4 compatibility library, it just leaves the instance set to whatever junk was originally in the buffer. That was the reason why propagation to our AFS kaserver environment wasn't working in pre-prod, because I started using our production AFS kaserver administrator instead of a custom one I'd created for testing.

I also rediffed a clean patch for MIT Kerberos 1.4.4 and fixed a few other minor issues. This code is going into production on Saturday.

You can get the latest version from the krb5-sync distribution page.

2007-08-08: kadmin-remctl 1.8

This is the first announced release of kadmin-remctl, although it's been available from my archives site for a while. This is a remctl front-end to the Kerberos kadmin functions to create, delete, enable, and disable accounts, change passwords, and so forth. We export this via a remctl interface for the use of other middleware at Stanford that can't cope with the direct kadmin protocol easily. We also provide a tool that will (after Saturday's upgrade) be based on remctl that the Help Desk uses to reset passwords for users, with various security checks.

This is the code that we're going into production with on Saturday and it works reasonably well, although it still leaves something to be desired in cleanliness and structure. For example, it uses the command-line OpenLDAP clients to modify Active Directory rather than using a native Perl LDAP implementation.

Hopefully, I'll have time to clean it up and improve it in the future. There's also a bit more work that needs to be done to support instance creation to work around iPass's completely broken security model.

You can get the latest version from the kadmin-remctl distribution page.

2007-08-09: Latest haul

There was a new Elizabeth Bear book out, so it was time for another Powell's order.

Elizbeth Bear -- Undertow (sff)
Ellen Kushner & Delia Sherman -- The Fall of the Kings (sff)
Ken MacLeod -- The Execution Channel (sff)
Sarah Monette -- The Virtu (sff)
Edward Tufte -- Beautiful Evidence (non-fiction)
Joss Whedon, et al. -- Serenity: Those Left Behind (graphic novel)

I also got a copy of Kushiel's Dart in hardcover, and I just ordered the second and third books in hardcover. I decided I really wanted to own the whole series in hardcover rather than paperback. I may well re-read them sometime soon, maybe after the second trilogy concludes.

2007-08-11: Kerberos upgrade

It's finally done, which means that the hardest part of the Kerberos project is done. Today, we switched our master realm from Kerberos v4 to Kerberos v5, which meant putting krb5-strength and krb5-sync into production, replacing our Kerberos v4 kadmind with a server that still handles examine, enable, and disable but proxies password changes to Kerberos v5, and replacing our kadmin remctl interface with kadmin-remctl. I also replaced the Help Desk password change client.

It all seems to have worked. Some of the systems (particularly the K4 authentication servers) are in a slightly weird state now, and lots of updated documentation needs to be written, but all the coordination, testing, and 27-step rollout is finished.

It's quite a relief. I'm now going to do as little as possible work-wise for the rest of the weekend, and then figure out what's next on the agenda.

2007-08-12: Glory Road

Review: Glory Road, by Robert A. Heinlein

Publisher: Avon
Copyright: 1963
Pages: 288

Pfc. Gordon, the first-person narrator who later picks the name Oscar, is a Vietnam vet given his discharge after being wounded. He heads for Europe, deciding to try for a degree from a European university paid for by some lottery tickets he won in a poker game. But while living cheaply (and mostly nude, Heinlein's regular obsessions showing in the background) on an island, waiting for lottery drawings, he meets a rather remarkable woman. And after discovering his lottery ticket is a fake, he answers an ad in the paper that seems targetted at him, meets her again, and ends up in a very different world.

The first part of this book is somewhat enjoyable. It doesn't have much of a plot, and it's full of Heinlein's typical off-hand political and social commentary, but it has a breezy, conversational tone that's quite readable. It helps that Oscar is self-deprecating and has a good sense of humor about his life.

Unfortunately, this only lasts until Oscar is taken into another dimension by the horribly attractive female, who he calls Star. He is hired to be a hero, for a great reward at the end of a dangerous mission. He gets led by the nose into various dangers, which he overcomes. Star manages to stay vague about exactly what that mission is, while bantering with Oscar in that annoying Heinlein fashion that reads like some combination of slang, excessive chivalry, jaw-dropping sexism, weird politeness, and exaggerated teasing.

And that's most of the book.

Oh, things happen. Oscar finds ways of defeating various monsters through quick thinking and ingenuity. He runs afoul of local customs by being unwilling to sleep with naked women who are parading themselves in front of him, in a pointed commentary on how dumb it is that Earth customs do not match Heinlein's sexual fantasies. And there is a great deal of annoying love story as Oscar falls for Star (he can't help it; she's declared to be irresistable by the author), subjects her to constant exaggerated compliments, threatens to spank her and treat her like a child, puts her firmly in her place a few times, and otherwise behaves like the typical sexist Heinlein hero. All of which is apparently horribly charming in her culture; either that, or she's so smitten with authorial fiat that she can't figure out he's acting like an abusive twit. As is sadly typical for these sorts of things, any real mutual understanding apparently pales in importance next to being the sort of hero and heroine who fall in love with each other.

But all of that is the standard sort of thing that happens around the plot, and there's basically no plot. They're on a quest to do something, about which Star is excessively coy but which eventually turns out to be stealing the Foozle from the Guarded Tower. (I think it was called something other than Foozle, but it was hard to care.) To the shock of absolutely no one, Oscar succeeds, with sufficient help from Star and her weird servant that one has to wonder what they needed him for. Indeed, it's so painfully obvious that Oscar is the least useful member of this party that even Oscar has to wonder what they needed him for. Upon asking that, he's given a remarkably silly response that doesn't actually answer the question.

After they find the Foozle, the book takes a sharp turn towards interesting for the last eighty pages or so. We finally find out what the hell is behind this adventure and why Star cares about the Foozle (although by "find out" I mean "are smacked upside the head with the Wet Trout of Exposition"). We get a bit more of Heinlein utopian government, which while remarkably impractical does manage to be sarcastic commentary on the governments we have. And we finally get a bit of character development as Oscar discovers some problems with life as an ex-hero. Unfortunately, that character development is rooted heavily in the relationship between Oscar and Star, and realistic relationship turmoil is not one of Heinlein's strong points; it's a battle between interesting and painful and painful mostly wins. But to give him credit, Heinlein does go for a somewhat non-traditional denouement and clumsily pokes at the happily-ever-after ending to see if it holds up.

The beginning and the ending both have the seeds of a good book. But, alas, the middle of Glory Road is page after page of this sort of thing:

"Oscar, by your standards — the way you have been raised — I am a bitch."

"Oh, never! A princess."

"A bitch. But I am not of your country and I was reared by another code. By my standards, and they seem good to me, I am a moral woman. Now... am I still 'your darling?'"

"My darling!"

"My darling Hero. My champion. Lean close and kiss me. If we die, I would my mouth be warm with your lips. The entrance is just around this bend."

"I know."

A few moments later we rode, swords sheathed and bows unstrung, proudly into the target area.

Best used for a drinking game in double entendre; within pages, you'll be blissfully unaware of just how bad the writing is. If you want to read 288 pages of that, by all means, have at it. I have a copy you can have.

Rating: 3 out of 10

Permanent review page

2007-08-13: krb5-sync 1.0

Last Saturday, I deployed all this code in production at Stanford, so it's no longer beta software. This releases cleans up some logging issues and filters out common errors so that I'm not deluged with e-mail from accounts created in Kerberos and not in Active Directory because they haven't yet been sponsored. It also fixes a few other minor logging issues.

You can get the latest version from the krb5-sync distribution page.

2007-08-17: Investing and betting

A recent article in Salon repeated the annoying comparison between investing and betting as a way of explaining the current mortgage collapse, which annoyed me enough to send a letter, and I'm apparently not done talking about this. Warning: this will be long.

I think calling investment betting is fundamentally broken. Worse than that, I think it betrays a fundamental lack of understanding of economic principles, and while that's to be expected from a lot of individuals, having a columnist whose job it is to explain subtle issues to people use the same broken analogy is disappointing and decreases the economic understanding of the readers.

You can stretch the definition of betting to mean any time you do something that may or may not make more money. Working hard to finish a project so that you get a bonus is in some sense "betting" that you'll finish the project and the work will be worth it, and sometimes people call it that. In that sense, investment (and many other things) is betting. But when someone calls investment betting, they don't just mean that. They imply it's like sports betting: it's speculation on some fundamentally meaningless activity, it's risky, winning or losing a bet doesn't do anything useful for the world, and it's vaguely shady. And, importantly, betting is something you can not do, and that indeed most people don't and shouldn't do. It's something people do for entertainment with money they can lose.

That last is what makes this analogy so broken. The implication is that investing (or at least some form of investment that the author doesn't like) is something that we can not do. And that's simply wrong. Our economy, any modern economy, would not survive without investment. Many of these investments that people complain about are natural consequences of markets; it would be almost impossible to prevent people from making these investments without draconian legislation, and passing such legislation would cripple things that free markets are quite good at.

Let's talk about some basics. Broadly speaking, you can divide investments into two basic types. Either you purchase something, or a share of something, that you think will retain its value and possibly grow in value (real estate, gold, stock in a company); or you loan money to someone who will pay you interest in exchange for the loan (bonds, treasury certificates, bank deposits).

Everything you can do with money that you're not spending, other than putting it all in your mattress, is investment. If investment is gambling, almost everyone in the world is gambling. Even depositing your money in the bank is investing. It's a loan to the bank, an investment of the second type. The bank pays you interest in exchange for the right to use your money (generally loaning it out to someone else at a higher rate of interest). It's a loan that, for most accounts, the bank promises to repay on demand whenever you want, but that's still a loan, just a fairly safe one.

This is where risk comes in. Risk is easiest to think about with loans. It is, simply, the chance that the person or entity to whom you loaned money will actually pay it back. We're all used to this: when you loan money to a friend, there's always a chance that they won't pay you back. You take that into account when you decide to make the loan. Just because there's a chance they won't pay you back, that doesn't mean that you're gambling; you're taking a "calculated" risk, which means that you're evaluating the level of risk and deciding that it's worth it to you.

Risk is a bit harder to think about when it comes to buying property, but think of buying a house. You're running various risks when you do that. The house may have some undisclosed structural problem that you'll then have to pay a lot of money to fix. Someone might build an airport next to the house, making the area very noisy and reducing the value of the house. Anything that you buy has this problem. Most things we buy to use and expect to throw them away when we're done. Houses are the main thing (outside of pure investments like gold or stocks) where we're used to thinking about both the use of the house and the value when we sell it later.

Obviously, risk is bad. Risk means you could lose something: the money of the loan, which is never paid back, or the value of the house, which was lost when someone built a trash dump next to it. The basic idea of an investment market is that when you take a risk, you should be compensated for it with higher profit if the risk pays out. This is for obvious reasons. Risk is bad, so unless people make more money from higher risk, they simply won't do things that are high-risk at all. If everyone always took the minimum possible risk, only people who had absolutely perfect credit with million-dollar incomes would be able to buy a house. We want people to take higher-risk activities occasionally; it's good for the economy, it lets average people buy houses, and it lets companies try things that may or may not work. So, in exchange, people are paid more for taking a higher risk, to give them some incentive for doing so. Someone who isn't a millionaire (more likely to not pay back a loan, and therefore is higher risk) pays higher interest to compensate the person loaning them money for taking a higher chance they won't be paid back.

Now, some people need all of their money and don't want to take any risk. They can deposit their money in federally-insured accounts, taking as low of risk as we can offer for an investment. In return, though, they get minimum income from that investment. People who take higher risk (buying stock, for instance, which runs the risk that a company may cease to be profitable and its assets won't be worth what the company is valued for on the market) get higher income.

And this is where we get to the part that people have the most trouble with, I think. Investments aren't just made and held all the time. They're bought and sold constantly, which can be confusing. But one of the major reasons for this is the notion of valuation. The basic economic principle at work is that you never really know how much something is worth until you sell it. Think of some family heirloom — you can have it appraised, you can guess, but you never know how much it's truly worth until you find someone who wants to buy it.

In order to make useful decisions about which companies are worth owning, or which loans are paying enough to be worth their risk levels, someone has to evaluate those companies or those loans and communicate how valuable they are. And you never know the results of that investigation until that person buys or sells for a particular price. That price reflects their belief of how much something is worth. Prices are communication of other people's knowledge.

An example. If you have a loan that, when held, returns 7%, and no one wants to buy it out for the price that you're offering, that means the market has decided that the risk level is high enough that 7% isn't enough. If you want to sell it, you'll have to sell it for less than the original value of the loan. That means that, for the new buyer, the loan will pay 8% of what they paid for the loan instead of 7%. See what happened? The process of selling the loan has adjusted the rate of return to match what the market decided the risk was. It fixed something that was inaccurate, and afterwards we have better information.

Markets are exceptionally good at this. In the long run, markets arrive at excellent estimates of exactly how much something is worth, or of exactly how much risk there is in a given loan. They do this through sheer self-interest on the part of the investers: if you can evaluate the worth of something better than other people, you buy things that are too cheap and sell things that are too expensive, and you make lots of money. And in the process of making all that money, you end up adjusting the market prices to be more accurate. This is the purpose of investment markets in a nutshell.

This is valuable and important work! If it weren't for investment markets, we would have no idea how much someone should have to pay for a loan given their chances of paying it back, or whether a company's business model is good enough to be worth investing in. The market enables millions of people to make decisions about their areas of expertise, collects all that information, and reflects it in prices with a high level of accuracy.

Sounds less and less like gambling, doesn't it? It is a little like gambling in that gambling odds serve the same purpose, and similarly arrive at very accurate estimates of probabilities. But unlike the gambling that people think of (sports, poker, blackjack), investments place a value on something that's real, and having that valuation helps the economy work better because the economy now has more information.

The normal, competent invester is doing something that looks nothing like gambling. They're buying investments to hold for the long term, based on their desired level of risk and profit. If they have a high risk tolerance, they buy things that have a high level of risk and pay a correspondingly high rate of return. If they have a low risk tolerance, they buy things like US treasury bonds that have a low rate of return but very little risk. It's really that straightforward.

Whenever you see people freaking out about investment markets or claiming that some investment is gambling, remember that the purpose of an investment market is to value something by buying and selling it and figure out what's being valued. For example, the recent market chaos is over home mortgages, specifically "sub-prime" mortgages (loans to people with bad credit). The securities that are being traded are essentially pieces of home mortgages, pieces of loans. We're discovering that many buyers in the past thought that these loans were lower risk than they actually are (probably because we've had an economic boom and an excess of borrowing that's now ending). Now, the market is doing what markets do and doing a mass revaluation of all those loans. Previous buyers are discovering that they were wrong about the value of the loans, or in other words that they weren't being paid enough interest on their investment to have the risk be worth it. New buyers are purchasing the loans at more accurate values (and hence higher interest rates for them). This shakeout is exactly what markets are for. At the end, we'll end up with a much more accurate view on what those mortgages are really worth.

Figure out what the market is putting a value on and suddenly the mechanisms make sense.

One final thing. There are certain types of investments that get an even worse bad rap, that people claim are just gambling. An example would be Dow Jones futures, where people buy and sell securities that represent the chances that the Dow Jones average will go up or down at the start of the next trading day. Isn't that gambling? Aren't people just making bets on their guesses of what the Dow will do, with no purpose outside of that betting?

No. Look at what this market is valuing. The Dow is a collection of stock in a bunch of companies (that happen to be considered the core companies of the economy in some sense). Stock in those companies only trades for a short period of time during each day. However, economic activity and news about companies doesn't stop when the market closes. What the futures market does is let people continue that valuation process while the market is closed, which provides useful information to the people in the market at the start of the next day and feeds in to the overall process of valuing all those companies and the economy as a whole. This kind of activity will happen. It can't not happen. People don't stop caring about the value of things while the stock market is closed. They will create ways to act on those opinions and additional information whenever they want, and the futures market in stock market indexes is part of that.

2007-08-25: Obvious security

It occurs to me, after a ton of fake YouTube links that appear to be the spam du jour, that quite a lot of phishing would be stopped completely by the simple expedient of all HTML-enabled mail clients looking at <a> tags. If the text of the anchor looks like a URL and that URL doesn't match the actual target URL, throw all sorts of bright red warnings all over the page, remove the link, and so forth.

I can't think of any drawbacks to this and a good 80% of phishing messages I see would be caught by that. And yet, I don't remember hearing about clients doing this. Why not?

2007-08-26: remctl 2.10

I finally found time this weekend to incorporate a bunch of additional work by Marcus Watts, including a Java client that speaks protocol v2 (yay!) and a Java server. I also fixed a ton of minor bugs that I'd been saying "next release" for. I really went rather too long between releases this time, so it's good to get a release out.

Expect more in the not too distant future, since several other schools seem to be deploying remctl and feeding me patches. Very gratifying, that.

You can get the latest version from the remctl distribution page.

2007-08-27: krb5-sync 1.1

MIT Kerberos uses an annoying structure for Kerberos principals. Rather than giving you nul-terminated strings, all the portions of a Kerberos principal are krb5_data structs with a length attribute. It is nice in that it means principals could contain nul characters if anyone wished, but it's hard to deal with in C.

It turns out that kadmin nul-terminates principal instances, and the previous krb5-sync versions were relying on that. However, when processing kpasswd protocol requests, it doesn't, which was causing root instances to not be propagated properly into Kerberos v4 at Stanford (and may be behind some of the crashes we were seeing in kadmind). My previous code for checking the instance for propagation also didn't cope with some cases where the instance was a substring of an allowed instance.

I rewrote the whole checking loop as a state machine to fix both of these problems, and now it seems to work.

You can get the latest version from the krb5-sync distribution page.

Last modified and spun 2017-09-17