rss logo

How to transfer a huge file through internet thanks to GNU/Linux

I recently have to transfer a big file between two GNU/Linux hosts and through a slow connection network.

I'll show here what I did to achieve it.

Network diagram

For the example I use a private ip address (192.168.1.200) but it obviously works with a public ip.

The operation will consist in dividing the big file into several small ones. So in case of a network failure we will be able to resume the transfer where it has stopped.

GNU/Linux transfer big file via ssh and rsync diagram

Installing prerequisites

  • In order to transfer our files we will need, on these two machines, to install rsync and openssh :
root@host:~# apt-get install rsync openssh-server

Split the file

  • As said before, we will divide the large file into several small ones thanks to the split tool :
  • split options :
    • -d : use numeric suffixes starting at 0
    • -b : files size, here we will divides our big file into 5M bytes files.
user@SOURCE:~$ split MyBigFile.mkv -d -b 5M
user@SOURCE:~$ ls -lh
-rw-r--r-- 1 std std 5,0M  8 déc.  23:47 x00
-rw-r--r-- 1 std std 5,0M  8 déc.  23:47 x01
[...]
-rw-r--r-- 1 std std 5,0M  8 déc.  23:48 x9481
-rw-r--r-- 1 std std 5,0M  8 déc.  23:48 x9482

Files transfer

Now we can transfer the files to our destination host. In case of failure the rsync software will be able to resume the transfer from the last file transferred and thus avoid starting from the beginning.

  • rsync options :
    • -a : archive mode
    • --bwlimit=40k : (Optional) We can set maximum bandwidth. Here, 40kB/s.
    • --rsh='ssh -p 22' : Specify the 192.168.1.200 ssh port.
    • --stats : (Optional) In the end, give some file-transfer stats.
    • --progress : (Optional) show progress during transfer.
    • user@192.168.1.200:/home/SOURCE/x* : Source folder, where our split files are. Here in /home/SOURCE/ directory.
    • /home/DESTINATION/ : Destination folder.
user@DESTINATION:~$ rsync --bwlimit=40k -a -v --rsh='ssh -p 22' --stats --progress user@192.168.1.200:/home/SOURCE/x* /home/DESTINATION/

Rejoining split files

  • Once transfer complete, it's time to rejoin split files :
user@DESTINATION:~$ cd /home/DESTINATION/
user@DESTINATION:~$ cat x* > glory.41.720p.hdtv.x264-verum.mkv
user@DESTINATION:~$ ls -lh glory.41.720p.hdtv.x264-verum.mkv
-rw-r--r-- 1 toi toi 2,9G  8 déc.  23:45 glory.41.720p.hdtv.x264-verum.mkv
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Contact :

contact mail address