When you’re uploading or downloading large files to or from a remote server, your connection speed will be limited by the maximum throughput of your Internet link. The rsync command-line utility can help you make the most of your available bandwidth by transferring files to and from the remote host in chunks as small as you specify, which means that Rsync can transfer files much faster than if it transferred the entire file before moving on to the next one. In this article, I’ll explain how to ,speed up file transfers using rsync with gnu parallel, and show you how to use them together to speed up file transfers on rsync multiple threads.
Backup your files first
Before you do anything, it’s always a good idea to backup your files. That way, if something goes wrong, you have a copy of the original data. There are many ways to backup files, but we recommend using rsync. rsync is a fast and efficient way to transfer files and can be used in conjunction with GNU Parallel to speed up the process. Here’s how rsync -rltp -nxt Ipsum * remote:/backup/incoming-local-directory/ gnu parallel –strip=2 ‘rsync -to -nxt Ipsum *.txt remote:/backup/incoming-local-directory/’ ::: ipsum1.txt,ipsum2.txt
Getting Started
rsync is a great tool for copying files between servers. It’s fast and efficient, but it can be even faster with the help of GNU Parallel. By using both tools together, you can speed up file transfers and get your work done more quickly. Here’s how to do it ;
– First, install GNU Parallel on both machines: sudo apt-get install parallel
– To do this, first copy the original file with rsync (e.g., rsync -av /source/path/ /destination/path). Then use parallel (e.g., parallel -j4 ‘rsync -av –delete-after /source/path/* /destination/path’) to copy all of the updated versions at once.
Create two lists of files – one for local source and one for remote destination echo file1 > list_local $ echo file2 >> list_local echo file3 >> list_local echo file4 >> list_remote # Copy the local files to the remote server:
rsync -av –delete-after –progress ./list_local ./list_remote/
Basic Example of rsync + gnu parallel
If you have a lot of files to transfer, using rsync in combination with GNU parallel can speed up the process. Here’s a basic example of how it works rsync -av –progress /source/directory /destination/directory
gnu parallel -j 4 –max-lines=1000000 rsync -av –progress /source/directory /destination/directory
Conclusions
rsync is a great tool for quickly copying files between two locations. However, it can be slow when copying large numbers of files. GNU Parallel is a great tool for speeding up file transfers by using multiple processors. When used together, these two tools can significantly speed up the process of copying files.
Leave a Reply