It appears that there is more than just “regular” process priority at work here. As I noted in the question, the task scheduler by default runs your task at lower than normal priority. This question on StackOverflow describes how to fix any task to run at Normal priority, but the fix still leaves one thing slightly different: memory priority. Memory priority was a new feature for Windows Vista, and is described in this Technet article. You can see memory priority using Process Explorer, which is a must-have tool for any administrator or programmer.
Anyway, even with the scheduled task priority fix, the memory priority of your task is set to 4, which is one notch below the normal setting of 5. When I manually boosted the memory priority of my task up to 5, the performance was on par with running the process interactively.
For info on boosting the priority, see my answer to a related StackOverflow question about IO priority; setting memory priority is done similarly, via NtSetInformationProcess, with PROCESS_INFORMATION_CLASS set to ProcessMemoryPriority (the value of this is 39 or 0x27). I might make a free utility that can be used to set this, if others need it and don’t have access to programmer tools.
EDIT: I’ve gone ahead and written a free utility for querying and setting the memory priority of a task, available here. The download contains both source code and a compiled binary.