Watch out for zombies!

My mother’s Yahrzeit is coming up, and her name will be on the Kaddish list this Shabbat, so perhaps it’s appropriate that I’m making a posting she would have considered complete gibberish.

For the last three weeks, my MacBook Pro has been giving me fits. When I tried to start a program, sometimes it just wouldn’t start. And, when I looked in /var/log/system.log, it was littered with lovely messages like these:

Apr 17 00:45:31 dssmac com.apple.launchd[103] ([0x0-0x2effefd].com.apple.systemevents): fork() failed, will try again in one second: Resource temporarily unavailable
Apr 17 00:45:31 dssmac com.apple.launchd[103] ([0x0-0x2effefd].com.apple.systemevents): Bug: launchd_core_logic.c:6780 (23714):35: jr->p
Apr 17 00:45:36 dssmac /usr/bin/osascript[13552]: spawn_via_launchd() failed, errno=12 label=[0x0-0x2f01eff].com.apple.systemevents path=/System/Library/CoreServices/System Events.app/Contents/MacOS/System Events flags=1
Apr 17 00:45:36 dssmac com.apple.launchd[103] ([0x0-0x2f01eff].com.apple.systemevents): fork() failed, will try again in one second: Resource temporarily unavailable
Apr 17 00:45:36 dssmac com.apple.launchd[103] ([0x0-0x2f01eff].com.apple.systemevents): Bug: launchd_core_logic.c:6780 (23714):35: jr->p
Apr 17 00:45:42 dssmac /usr/bin/osascript[13553]: spawn_via_launchd() failed, errno=12 label=[0x0-0x2f03f01].com.apple.systemevents path=/System/Library/CoreServices/System Events.app/Contents/MacOS/System Events flags=1
Apr 17 00:45:42 dssmac com.apple.launchd[103] ([0x0-0x2f03f01].com.apple.systemevents): fork() failed, will try again in one second: Resource temporarily unavailable
Apr 17 00:45:42 dssmac com.apple.launchd[103] ([0x0-0x2f03f01].com.apple.systemevents): Bug: launchd_core_logic.c:6780 (23714):35: jr->p
    

with the occasional

Apr 15 14:41:42 dssmac kernel[0]: proc: table is full    
    

thrown in for bad measure.

I couldn’t figure out what was going wrong (Activity Monitor only showed between 60-80 processes, far fewer than the system limit), so yesterday, I reinstalled Mac OS X (using the archive-and-install method) — and it didn’t help.

I was, needless to say, unhappy. I hadn’t brought my external drive to the office, so I couldn’t do a bare-metal reinstall yet. But I could (and did) tweet about my problem:

Still getting fork(1) failures (“resource not available” — which one, dammit?), so I guess it’s time for a full reinstall. Crud.

This one caught the eye of many people who wanted to help, and I want to mention two in particular:

Ed Costello thought it might be hardware — I ran the hardware diagnostics, which showed nothing.

Rich Berlin (from Sun) made the suggestion which wound up putting me on the right path — he suggested running:

sudo dtrace -n 'syscall::fork*:entry{printf("%s %d",execname,pid);}'

which showed two Eclipse-based processes forking their little hearts out. So I did a “ps” to discover what they were (unsurprisingly, Lotus Notes and Lotus Sametime), but what startled me was how many “(NotesDynConfig)” processes there were in the process table. I wondered how many, so I ran

ps -aA | wc

and was shocked to see a result of about 160, compared with the 70 processes shown in Activity Monitor. So I stopped Notes and suddenly, I was down to 70 processes via both methods.

It seems that Activity Monitor doesn’t report zombie processes. Neither does the line at the top of “top(1)”, which I’d also used while trying to troubleshoot.

Given that discrepancy, I can now understand why the system was running out of processes. I don’t know why Notes is leaving zombies around, but that’s a problem for another day (my next step is to upgrade to the latest beta and see if it helps — I’ve also reported the problem, of course).

And I guess I probably don’t really have to do a full reinstall…though I might, anyway — it’s my Windows training coming to the fore.

Twitter Search beats Google — malware attack averted

As I was driving to work this morning, my iPhone tinged, letting me know I had a new SMS awaiting me. And it tinged a second time as I pulled into my parking place, since I don’t check SMS messages while I’m driving.

It was a Facebook notification from an IBM colleague with a subject of “How did you manage to get on this video?”, sent to me and 19 others, with a link to a geocities.com page.

I was immediately suspicious, because the note wasn’t in my colleague’s style — but it was rather short, so perhaps that wasn’t valid. I was also suspicious because the names on the note were a rather mixed bag.

But it was vaguely possible that the video had something to do with IBM’s Smarter Planet initiative, so I didn’t want to discard the note.

Instead, I did the obvious thing: I Googled for “Facebook” and “get on this video”, looking for reports of malware. But I found nothing. I tried a few other variants, including “Facebook malware” and still found nothing.

So I went to plan B: Twitter. Nothing was obvious on my home page, so I posted a query: “Just got suspicious-looking facebook msg: ‘How did you manage to get on this video?’ with a link to GeoCities. Anyone know if it’s malware?”

While I waited for an answer, I tried Twitter Search, using “Facebook” as my query. Within seconds, I had my answer — yes, it was malware, and apparently virulent stuff.

And when I went back to my Twitter page, I’d gotten three replies from friends telling me the same thing (the first one arrived less than a minute after my tweet).

For timely questions, Twitter is my new go-to tool — sure, Google has depth, but it’s not instantaneous. Twitter gives me three paths to an answer:

  • Stumbling on it in my friends’ tweetstream without ever asking the question
  • Asking the question and hoping a friend answers
  • Using Twitter Search

My search strategy on Twitter is different than what I’d use on Google, though. On Google, it helps to be specific — a search on “Facebook” alone would be pretty useless, hence my attempts to qualify with the hook phrase and the word “malware”.

In contrast, on Twitter, timeliness is your friend — a one-word query (“Facebook”) is just fine, because you’re going to get the current conversation, and the human eye can do a good job of picking out the pay dirt if there is any.

I guess I’ll never find out how I got on that video, though.