A failure predictable at 2Gb sounds like the target filesystem is to blame… Are both on NTFS? Are you piping through any compression (zip used to fail at 2gb boundaries) ((is apache doing compression)) I have copied many files over 20Gb using robocopy (as others have mentioned) but I’d avoid using the /MIR switch until … Read more
If you really want to use rsync, it sounds like you’ll need some combination of –backup, –backup-dir, and –suffix. The closest I think you could get is with something like this rsync -abv –suffix R1 –remove-source-files src/ dst/ This would do close to what you want, but it would not rename the files exactly the … Read more
Quoting man ssh (which is the base used by scp): Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. The problem is that compressing the data takes more time then just sending it over the network.
The “Binary” transfer mode of FTP copies files exactly, byte for byte. Simple and straightforward. When bringing text files between different operating systems, though, this might not be what you want — different operating systems use different codes to represent line breaks. The “ASCII” mode exists for this purpose: it automatically translates all line endings … Read more
rsync I’d use rsync before I used ftp or tftp. More options and (in my experience) more reliable transfer.
If there’s an rsync option to do exactly what you’re asking, I didn’t find it in the manpage either. 🙂 That said, I don’t see the problem with grepping the output of rsync -i to parse out exactly what you need. That feels nice and Unixy to me. One nit-picky quibble with your rsync command: … Read more
SSH is much more tolerant of network security devices like firewalls and things that do NAT. FTP is famously tricky to firewall, and generally requires one end-point to have a real IP addresss (i.e. no NAT). SSH is better at handling NAT. In fact, both ends can be behind NAT which is generally not possible … Read more
Searching for “high latency file transfer” brings up a lot of interesting hits. Clearly, this is a problem that both the CompSci community and the commercial community has put thougth into. A few commercial offerings that appear to fit the bill: FileCatalyst has products that can stream data over high-latency networks either using UDP or … Read more
This might be what you’re after. % rsync -avvz –times –stats –checksum –human-readable –acls \ –itemize-changes –progress \ –out-format=”[%t] [%i] (Last Modified: %M) (bytes: %-10l) %-100n” \ /usr/include/glib-2.0 my-glib-copy/ The switches breakdown as follows: -avvz = archive, verbose x 2, compress –times = preserve modification times –stats = give some file-transfer stats –checksum = skip … Read more
If /foo/bar is on NFS (or possibly some FUSE filesystem), that might be the problem. Either way, adding -O / –omit-dir-times to your command line will avoid it trying to set modification times on directories.