I actually had a problem with my 7800GTX after going from an ATi card back in the day. My framerate was 30% off in 3D mark compared to similiarly configured systems. Turned out I had run into a driver problem. My recommendation is that anyone who upgrades their video card, run 3D mark and compare to similarly configured systems. Keep in mind that a lot of people cheat at that and most overclock, so just google your config and 3D mark scores for an accurate comparison from people who post on forums.
Anyway, having said that, things have changed with Windows 7/Vista and in this day and age, there is nothing to worry about. Driver uninstalls are comprehensive.
My recommendation:
1. Start your system in safe mode.
2. Uninstall the Nvidia control panel, the nvidia phys x stuff and drivers. Do the control panel first.
3. Restart your computer again in safe mode. Go to the windows control panel to see if both components are gone. Sometimes you can't uninstall both together. So you may have to uninstall one of those this time. If there is nothing, good, shutdown your computer. If something still exists, uninstall and restart again in safe mode till everything is gone, and then shutdown.
Once you've shutdown, put in your new card, and go to windows as normal. Don't go to normal windows i.e. non-safe mode windows after uninstalling everything with your old card, or it will autodetect and you'll be back to the start.
Anyway, it might sound a little complex, but it is pretty simple. Only install the new ATi drivers when you've uninstalled Nvidia stuff in safe mode. You should have no problems, in this day of computers.
edit:
Yea I should mention that PC Gamer discussed this with one of the Maximum PC guys and Logan Decker on a podcast a few months ago. They said that if you really paranoid, then you should take the steps (I mentioned above), but in this day in and age you'd even be crazy to format... even when going from one company to another. They too recommend a simple uninstall, or a driver cleaner as GPW said.
