Updated 06-Aug-2013 : Script updated to transfer multiple files in single command [ jump to this ]

Updated 14-Oct-2013 : Minor performance improvements in script [ jump to this ]

Updated 01-Nov-2013 : Better command line options, better error reporting [ jump to this ]

Updated 05-Feb-2014 : Upgrade script to use netcat6, remove dependency on ‘screen’, auto compression for text files [ jump to this ]

I frequently transfer several GB’s of data between my work and home laptops. Pendrives are useless because of the amount of data that I have to transfer and their speed is unbearable. That is when I discovered that netcat and tar is one of the fastest ways to transfer data across a network. The problem with this is that it requires one command to be run on the source machine and one command to be run on the destination machine. When was the last time you ran a command on the destination machine just to copy a file from the source. This is cumbersome. So, I designed this bash shell script which makes the process of using netcat and tar for file transfer almost similar to scp or rsync on linux. So, there is no need to run any command seperately on the source and destination, just one command on the source side to do the job.

This script is not perfect… Currently, it is only able to transfer a single directory (all contents of directory) or a single file in one invocation of the script. I am working on making it as close as possible to scp command on linux. Do check back on this page for updates.

For those wondering how I transfer files over the network between two computers running Linux, here it is. Connect the two laptops via ethernet cable. Then, under IPv4 settings for ethernet connection, change method to “Manual”. Then under “Addresses”, click “Add”. For computer A, enter (Address, Netmask) as (10.0.0.1, 255.255.255.0) and for computer B (10.0.0.2, 255.255.255.0). Save your settings. Unplug and plug in the ethernet cable on any one computer and you should automatically have ethernet connection established.

Now, let us enable passwordless ssh login from one computer to the other so that you don’t have to enter the password of the other machine every time you use this script. On computer A, run

$ ssh-keygen -t rsa

Press ‘Enter’ for all questions.

$ ssh-copy-id username_on_computer_B@10.0.0.2

You will be prompted for a password. Enter the password of the user “username_on_computer_B” on Computer B. Next, test whether you are able to login to Computer B without password.

$ ssh username_on_computer_B@10.0.0.2

The above command should not prompt you for password. If this step is done, perform the same exact steps on Computer B, as follows.

$ ssh-keygen -t rsa
$ ssh-copy-id username_on_computer_A@10.0.0.1
$ ssh username_on_computer_A@10.0.0.1

Now, passwordless login should work for computer A to computer B and vice versa.

Now, you are ready to use the bash shell script to perform a netcat+tar file transfer at blazing speeds.

Oh, almost forgot, your computers will require some additional software not installed by default.

On source and destination computers, you will need to install netcat, pv and screen, like below (Ubuntu)

$ sudo apt-get install netcat pv screen

Now, you are surely ready to use the shell script as follows

$ cd ~
$ wget http://pastebin.com/raw.php?i=HUqz7u1y -O ncp.sh
$ chmod +x ncp.sh
$ ./ncp.sh source_dir_or_file username_on_computer_B@10.0.0.2:dest_dir 0

The first argument is a single source directory or a single source file on Computer A that you want to transfer to computer B. The second argument is the username, ip_addr and target folder of the destination machine(Computer B in this case) which is similar to the syntax used in scp or rsync. The last argument tells whether or not to use compression while transferring files (1 to use compression, 0 to not use compression).

The last argument (enable or disable compression) is a very important that affects data transfer speeds and time taken for data transfer. For video or music files which are already compressed, there is no point in enabling compression in the command. This will simply increase CPU usage to compress data that cannot be compressed much more and thus will decrease network transfer speeds. However, you can enable compression while transferring text files (which can be compressed massively) thus yielding a smaller amount of data that needs to be transferred which inturn decreases transfer time. Without compression, you should see about 100Mbps of transfer speeds indicated over ethernet.

Let me know if you think of any useful additions to the script or anything that can be improved.


UPDATE (06-AUG-2013)

The script has been updated to transfer multiple src files and/or directories to the destination in a single invocation of the command.

So, you could use the command like below

$ ./ncp.sh src_file1 src_file2 ... username_on_computer_B@10.0.0.2:dest_dir 0


UPDATE (14-OCT-2013)

Some minor performance improvements when transferring multiple files at one go using the script. Transferring the shell script that set’s up netcat on the remote machine only once and using the same script the next time. This should save on the time needed to a scp transfer of the shell script.

The new version of the script can be obtained using

$ wget http://pastebin.com/raw.php?i=0WszhTKh -O ncp.sh

All the other options remain the same as before.


UPDATE (01-NOV-2013)

I have updated the ncp shell script with better command line options handled using getopt, so you no longer need to explicitly specify a ‘0’ at the end of the command to disable compression. Compression is now disabled by default for the command, and you only need to give the ‘-c’ option to the command if you want to enable compression. There is also a new man page like help that you can get with the ‘-h’ option.

So, here is how to use the latest version of the ncp script.

Download latest version of ncp script

$ wget http://pastebin.com/raw.php?i=c4yVeA62 -O ncp.sh

View command usage with

$ ./ncp.sh -h

Data transfer with compression enabled

$ ./ncp.sh -c src_file1 src_file2 ... username_on_computer_B@10.0.0.2:dest_dir

Data transfer without compression

$ ./ncp.sh src_file1 src_file2 ... username_on_computer_B@10.0.0.2:dest_dir




UPDATE(05-FEB-2014)

I have updated the ncp script to use netcat6 (nc6). netcat6 can be installed in Ubuntu with the following command.

$ sudo apt-get install netcat6

As a result of moving to netcat6, you no longer need to have ‘screen’ installed on the destination machine to use this script.

Also, when the script is run without the compression option (i.e, without -c option), the script tries to automatically detect the type of file that is being transferred (using the linux ‘file’ command) and auto enables compression only if it is a text file (as text files usually benefit the most by compression). Thus, when the script is run without any options, it provides arguably the best possible data transfer times (by enabling compression for text files and disabling compression for other types of files) automatically. Ofcourse, if the -c option is provided, then every file that is transferred will be compressed.

To download the latest version of the ncp script,

$ wget http://pastebin.com/raw.php?i=VyHeasH0 -O ncp.sh

Usage of the shell script remains same as the previous version.