Global distcc is a no no!

I ran into a REALLY frustrating problem last night.  I recently updated my Ubuntu install to Karmic (fresh install), and last night a bunch of updates installed.  These updates included a new kernel, a new version of gcc, and several other things.  Of course, it asked me to reboot since a new kernel was installed.  I rebooted my computer, not thinking anything about it, only to be greeted by an error telling me that it couldn’t load my NVidia modules.

At first, I was like, “What the heck!  Dkms didn’t rebuild my graphics driver!”.  My first instinct was to `dpkg-reconfigure nvidia-glx-185`.  As soon as it got to the point of rebuilding the kernel module, it failed and gave me a log file to look into.  The log file said that the build process failed a cc sanity check, and that “The C compiler ‘cc’ does not appear to be able to create executables”.  I immediately tested to see if a simple .c file would compile with gcc, and it did.

I did a lot of google searching, and this phrase showed up several times in a lot of different contexts.  From that point, I started asking myself what I have done to my system that might be considered “non-standard”.  Then I realized that I had set $PATH to include the distcc/ccache version of gcc before the normal versions globally in /etc/environment.  I tried manually setting my path and rebuilding the module that way, but it still failed.  When that didn’t work, I removed the extra path elements from /etc/environment, rebooted, and the module built fine.  Dkms must not have inherited my root prompt’s environment.

Lesson learned: do NOT set distcc/ccache paths globally…do it per user.  If you have a lot of users and you don’t want to set it for each one, try setting it in /etc/profiles (which supports real bash scripting) based on a certain group or uid’s >= 1000.

Here’s the question though…why did distcc cause it to fail?  I still can’t really figure this one out.  Granted, upgrading gcc on my computer made it incompatible with the version of gcc on my server (to which distcc jobs are distributed) but I hadn’t specified that it should use more than one thread.  The only thing I can think of is that dkms automatically uses two threads or something, and one of the files was trying to compile on the server.  This should not have caused the build to fail, since distcc should have just refused to compile files on the server; however it might have changed the return status to something indicating an error.  I’m not really sure how to test this theory to prove/disprove it.

Either way, I learned a valuable system administration lesson.  Learn from my mistakes…I’m good at making them.  😉


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: