We did this internally several weeks ago for security reasons. I was running IE with higher levels, but noticed that something was occurring.
You'll have to keep IE installed, though. The patching system uses it. Also, some sites won't work in other browsers with proper security settings. On the other hand, I dont't believe that once the new XP patch is released those sites will work either.
I'm currently using Mozilla, no plugins. Acrobat works fine. Also we don't use Outlook - too risky.