Updated: Never — Philip's Blog

Looking through my calendar (yes, I have one) for 2010 this morning, I discovered that I will be proctoring quite a number of BSD Certification sessions this year. Plenty of opportunity for people (you?) to sign up!

• 7 February at FOSDEM 2010
• Sometime during AsiaBSDCon 2010
• 6 May at NLUUG

Chances are there will be other opportunities throughout the year too.

Someone -- I thought it was Kristof, but he claims not to have this problem so it must be someone else -- told me a while ago that Telenet's DHCP server "exhibits weird behaviour". That sort of mystery certainly gets the hyperactive mind interested.

For totally unrelated reasons, I found myself looking at a packet capture of DHCP traffic on a Telenet connection. Indeed, there was something very strange in there. The DHCP client would get a perfectly fine lease with perfectly reasonably renewing and rebinding times. When the renewing timer (T1) expired, the client would unicast a DHCPREQEST to the server and expect a unicast DHCPACK back. Only the DHCPACK would never arrive, and the client would retransmit the unicast DHCPREQUEST messages until the rebinding timer (T2) expired. At that time, the client would broadcast a DHCPREQUEST after which the DHCPACK would arrive.

The fact that the DHCPACK messages came through the DHCP relay server put me on a side-track briefly. I discovered that the DHCP server (mentioned in option 54) would not respond to my DHCP requests. While it makes perfect sense to protect a DHCP server from clients, you do want your clients to be able to get packets to them somehow.

I sent some packet captures to a contact inside Telenet (thanks ;-) I couldn't imagine trying to explain this to a helldesk!) wondering if they'd put too sharp an access control list between me and the DHCP server (recently -- because I hadn't seen the problem before). After some digging, they found that I was sending my unicast DHCPREQUEST messages with a random source port number. From my reading of the RFC, this is "allowed", but no one else does it. It turns out that Telenet does some sanity checking (sensible precaution) on DHCP messages before allowing them to go to the DHCP server. This sanity checking does not like (or recognize, presumably) DHCP messages with a source port other than bootpc (68).

FreeBSD's dhclient is a rather old version of ISC's reference implementation, simplified by OpenBSD. I found that OpenBSD has had a patch for a couple of years that purported to fix this behaviour. When I ported this patch to FreeBSD however, I found that sendmsg would return EINVAL, which was not documented to ever happen.

Again I wondered how people without source code to their operating systems get through the day? Do they resort to alcohol and panic at this stage? I used DDB to set a breakpoint on sendmsg and stepped through briefly, expecting it to blow up somewhere quickly when copying in the iovec or so. No such luck however, and I found myself in sosend_generic, which is not so much fun to step through without symbol information, so I set up remote debugging so I could use ddd.

Eventually, I found my way to rip_output and found that my EINVAL came from here:

if (((ip->ip_hl != (sizeof (*ip) >> 2)) && inp->inp_options)
    || (ip->ip_len > m->m_pkthdr.len)
    || (ip->ip_len < (ip->ip_hl << 2))) {
        INP_RUNLOCK(inp);
        m_freem(m);
        return (EINVAL);
}

Oh dear...:

(gdb) p m->M_dat.MH.MH_pkthdr.len
$6 = 328
(gdb) p ip->ip_len
$7 = 18433

Obviously (to the trained -- or strained -- eye which sees this kind of thing often), 18433 and 328 are strikingly similar. Indeed - it helps if you put the bytes in the right order!

For hysterical raisins, the raw socket interface on BSD-derived network stacks expects the ip_len field of the IP header included when IP_HDRINCL is sent to be in host byte order. dhclient used to only send packets with headers through the BPF, which will put the packet on the wire exactly as given (ie: the ip_len needs to be in the right order). For reasons which don't seem to be explained in CVS history, OpenBSD decided to change this behaviour in their network stack (making it differ from every other network stack and many books written about sockets).

To make a very long story short: I committed revision 198352 to make dhclient on FreeBSD work in networks which put sharp teeth between DHCP clients and servers. Debugging the problem also kept me out of trouble for a couple of hours.

I'm told that finding the cause of weird errors in the protocol stack is now significantly easier with DTrace. I will have to find some time to play with that. While ddd "works", it's not exactly the most pleasant tool to work with.

Entirely aside: I'm still not convinced that "sharp teeth" should care about the source port of unicast DHCPREQUEST messages, but I'm happy to accept that if everyone uses port 68, there's no reason to gratuitously differ from that. Thanks to the Telenetists for helping me look into this.

10:25 BST

On Saturday at 15:00 local time, I'm proctoring the BSD Certification exam in Cambridge. So far, fewer people have signed up than the number of papers I have with me. If you're interested in taking the exam and you happen to be in the area for EuroBSDCon, you can still sign up on the website.

First come, first served!

19:54 JST

Just in case people are reading the Planets more than the conference websites, I thought I'd spam this here. :-)

If you are interested in taking the BSD Certification exam at AsiaBSDCon 2009 in Tokyo this Saturday (14 March 2009), please sign up for it on the website. I still have five exams "too many" if people want to sit them.

It would be silly to make empty exams travel 10 000km... :-)

Sign up now.

21:39 PST

More meetBSD devsummit at Google today. I spent most of the day kicking the Coverity nightly builds back into shape. They've been down for 18 months, mostly waiting for someone to put on the "okay, I'll do it" hat and actually do it. So that's me. I know, I'm a static analysis fanboy.

After the devsummit, I took plosher up on an offer to take a tour of ISC's datarooms in Redwood City. I also took along a todo-list of FreeBSD boxes we've got there to see if I could poke them into doing something more meaningful than suck power and distress airco. The ultimate geek experience!

I saw (part of) the wayback machine. It's red. :-)

Off to Boston very early tomorrow morning. I'm taking Caltrain from Mountain View to SFO at 05:30. That should put me in just the right mood for a six-hour flight with Untied. What fun.

I had a great time at meetBSD and the devsummit again. A very productive time too. As usual, I was very good to see everyone in meatspace again, and add some new face-to-name mappings along the way. Next time, Tokyo I think.

10:24 PST

I was fairly productive at the devsummit yesterday. Catching up on core email and trying to get the Coverity stuff going full steam again. I have a feeling there is still some paperwork between me and getting all the necessary super powers, but I'm happy to go through that.

Dinner last night at Paul's. I finally met the infamous dogs! I have added Robert to my list of people to trust with finding food. Delicious Indian food. Lots of it.

Much merriment when I explained how Linux "oops" worked. It made Peter cry. I still can't believe most Linux people run with "panic on oops" turned off. They must be nuts. Or on very bad drugs.

Today I plan to spend some more time in the socket code. It shouldn't be too difficult to do what we want to do.

14:42 PST

As expected, my talk came out a bit shorter than I expected. I aimed for about forty minutes-ish and I finished in just about 25 minutes. I got about fifteen minutes of very good questions though, so it worked out fairly well in the end.

I've put my slides online here, if anyone is interested. I plan to spend some time perhaps on my next flight fleshing it out with more interesting numbers and such.

09:42 PST

Yesterday was the first day of MeetBSD at Google. Finding the right building was an interesting experience involving many u-turns (a Californian speciality) after that, things went fairly smoothly.

I still need to work on my talk a bit. It turns out I'm talking about crypto acceleration. I wish I had found this out before I got on my plane and not after. I have a feeling my talk is going to be a bit hand-wavy and probably come out a bit short.

Last night, we celebrated the 15th birthday of FreeBSD with cake at the Zen Lounge. Kirk did an excellent job of carving up the cake. Sadly, I didn't bring my camera with me. I'm sure a number of good pictures will find their way to the interwebnets later on.

A number of us left when the music starting getting a bit too loud. We survived a ride from Peter. No christmas trees were encountered.

No hangover this morning. American beer... Though I did find some surprisingly tasty stuff. Completely lacking in alcohol, however.

Despite getting home at a civilized hour Sunday night, thanks to a ride from a colleague, I couldn't find my voice yesterday morning and spent most of the day in bed. BSD conferences are very bad for one's sleep-wake cycle.

I had a very good EuroBSDCon, as usual. My (Warner's) talk was a bit more hand-wavy than I had intended it. I discovered that I had run through almost half my slides in ten minutes talking at a rate of about 0.7 rwatson. Slowing down a bit I managed to finish the talk in about 45 minutes.

Some very interesting questions from the audience. Looks like people are getting interested in embedding BSD. People are starting to realize how dangerous GNU really is.

I'm looking forward to MeetBSD in California next month. I expect the weather to be a bit warmer there than in Strasbourg. :-)

I spent a significant part of yesterday on the train to Strasbourg. It was not a TGV so there were only two power outlets. I had to fight with a besuited gentleman over the one nearest to me.

Slept briefly last night after spending some time at the "Académie de la Bière". I had some excellent French beers most of whose names I've predictably forgotten.

The devsummit is well underway. We're all talking together on IRC. :-)

There is coffee.