By the look at the variables you’ve set, it seems like you are mostly concerned with write performance and do not care about possible data losses due to power outages.
You only will ever get the option for lazy writes and the use of a writeback cache with asynchronous write operations. Synchronous write operations require committing to disk and would not be lazy-written – ever. Your filesystem might be causing frequent page flushes and synchronous writes (typically due to journalling, especially with ext3 in data=journal mode). Additionally, even “background” page flushes will interfere with uncached reads and synchronous writes, thus slowing them down.
In general, you should take some metrics to see what is happening – do you see your copy process put in “D” state waiting for I/O work to be done by pdflush? Do you see heavy synchronous write activity on your disks?
If all else fails, you might choose to set up an explicit tmpfs filesystem where you copy your backups to and just synchronize data with your disks after the fact – even automatically using inotify
For read caching things are significantly simpler – there is the fcoretools fadvise utility which has the --willneed parameter to advise the kernel to load the file’s contents into the buffer cache.
Edit:
vm.dirty_ratio = 70
This should in theory give us 16GB for caching I/O and wait some
minutes until its writing to disk.
This would not have greatly influenced your testing scenario, but there is a misconception in your understanding. The dirty_ratio parameter is not a percentage of your system’s total memory but rather of your system’s free memory.
There is an article about Tuning for Write-Heavy loads with more in-depth information.