Although you should have no problem running a 2005 instance of the database engine beside a 2008 instance, The tools are installed into a shared directory, so you can’t have two versions of the tools installed. Fortunately, the 2008 tools are backwards-compatible. As we speak, I’m using SSMS 2008 and Profiler 2008 to manage my 2005 Express instances. Works great.
Before installing the 2008 tools, you need to remove any and all “shared” components from 2005. Try going to your Add/Remove programs control panel, find Microsoft SQL Server 2005, and click “Change.” Then choose “Workstation Components” and remove everything there (this will not remove your database engine).
I believe the 2008 installer also has an option to upgrade shared components only. You might try that. Good luck!