How to transfer a huge file through internet thanks to GNU/Linux

I recently have to transfer a big file between two GNU/Linux hosts and through a slow connection network.

I'll show here what I did to achieve it.

Network diagram

For the example I use a private ip address ( but it obviously works with a public ip.

The operation will consist in dividing the big file into several small ones. So in case of a network failure we will be able to resume the transfer where it has stopped.

GNU/Linux transfer big file via ssh and rsync diagram

Installing prerequisites

root@host:~# apt-get install rsync openssh-server

Split the file

user@SOURCE:~$ split MyBigFile.mkv -d -b 5M
user@SOURCE:~$ ls -lh
-rw-r--r-- 1 std std 5,0M  8 déc.  23:47 x00
-rw-r--r-- 1 std std 5,0M  8 déc.  23:47 x01
-rw-r--r-- 1 std std 5,0M  8 déc.  23:48 x9481
-rw-r--r-- 1 std std 5,0M  8 déc.  23:48 x9482

Files transfer

Now we can transfer the files to our destination host. In case of failure the rsync software will be able to resume the transfer from the last file transferred and thus avoid starting from the beginning.

user@DESTINATION:~$ rsync --bwlimit=40k -a -v --rsh='ssh -p 22' --stats --progress user@* /home/DESTINATION/

Rejoining split files

user@DESTINATION:~$ cat x* > glory.41.720p.hdtv.x264-verum.mkv
user@DESTINATION:~$ ls -lh glory.41.720p.hdtv.x264-verum.mkv
-rw-r--r-- 1 toi toi 2,9G  8 déc.  23:45 glory.41.720p.hdtv.x264-verum.mkv
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Contact :