Removing it is just fine; it’s unversioned and all of the text is stored in the SVN log anyways. If your commit fails, you can reuse it by calling:
svn commit -F svn-commit.tmp
subversion will then automatically remove your svn-commit.tmp
file.