Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
stalled while copying large files
06-12-2018, 04:17 PM (This post was last modified: 06-12-2018 04:38 PM by cchen.)
Post: #1
stalled while copying large files
Sometimes when we are copying resources across hosts, the process will be stalled and cannot go on, especially when the file is large and network is not functioning well. Here are some solutions to this problem:

P1). rsync

using rsync instead of scp. It's a bit more versatile in my opinion, e.g. it keeps track of which files are already there, so if the connection ever does break it can pick up from where it left off - try the --partial flag too - among other things.
Code:
rsync -avz --progress local/path/some_file usr@server.com:"/some/path/"
refer to rsync for more info

P2). Limit speed
setting a speed limit reduces the delays which make the TCP connection stall - at the expense of a slower transfer of course:
Code:
scp -l 8192 local/path/some_file usr@server.com:"/some/path/"

P3). Compression option
scp's -C option can speed up the transfer, reducing the probability that the transfer stalls.

P4). Disabling TCP SACK
Code:
sudo sysctl -w net.ipv4.tcp_sack=0

P5). use localhost as an "intermediary":
if you are copying things between 2 VMs, try to first copy to your localhost then from localhost to another VM

P5). avoid cross-border transfer

RR rayluk
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: