Your Own Linux

Tech Blog

  • Home
  • About
  • Blog
  • Technology
  • Business
  • Contact Us

How To Speed Up File Transfers Using Rsync With GNU Parallel

August 31, 2022 by Skyler Leave a Comment

When you’re uploading or downloading large files to or from a remote server, your connection speed will be limited by the maximum throughput of your Internet link. The rsync command-line utility can help you make the most of your available bandwidth by transferring files to and from the remote host in chunks as small as you specify, which means that Rsync can transfer files much faster than if it transferred the entire file before moving on to the next one. In this article, I’ll explain how to ,speed up file transfers using rsync with gnu parallel, and show you how to use them together to speed up file transfers on rsync multiple threads.

Backup your files first

Before you do anything, it’s always a good idea to backup your files. That way, if something goes wrong, you have a copy of the original data. There are many ways to backup files, but we recommend using rsync. rsync is a fast and efficient way to transfer files and can be used in conjunction with GNU Parallel to speed up the process. Here’s how rsync -rltp -nxt Ipsum * remote:/backup/incoming-local-directory/ gnu parallel –strip=2 ‘rsync -to -nxt Ipsum *.txt remote:/backup/incoming-local-directory/’ ::: ipsum1.txt,ipsum2.txt

Getting Started

rsync is a great tool for copying files between servers. It’s fast and efficient, but it can be even faster with the help of GNU Parallel. By using both tools together, you can speed up file transfers and get your work done more quickly. Here’s how to do it ;

– First, install GNU Parallel on both machines: sudo apt-get install parallel

– To do this, first copy the original file with rsync (e.g., rsync -av /source/path/ /destination/path). Then use parallel (e.g., parallel -j4 ‘rsync -av –delete-after /source/path/* /destination/path’) to copy all of the updated versions at once.

Create two lists of files – one for local source and one for remote destination echo file1 > list_local $ echo file2 >> list_local echo file3 >> list_local echo file4 >> list_remote # Copy the local files to the remote server:

rsync -av –delete-after –progress ./list_local ./list_remote/

Basic Example of rsync + gnu parallel

If you have a lot of files to transfer, using rsync in combination with GNU parallel can speed up the process. Here’s a basic example of how it works rsync -av –progress /source/directory /destination/directory

gnu parallel -j 4 –max-lines=1000000 rsync -av –progress /source/directory /destination/directory

Conclusions

rsync is a great tool for quickly copying files between two locations. However, it can be slow when copying large numbers of files. GNU Parallel is a great tool for speeding up file transfers by using multiple processors. When used together, these two tools can significantly speed up the process of copying files.

Filed Under: Technology

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

About Us

Hello there! I’m Skyler a professional tech guy you can count on. I’m here to share with you my broad knowledge when it comes to tech stuff. I do some configuring and coding that might be helpful to you. Read More…

Connect With Us

  • Facebook
  • Instagram
  • LinkedIn
  • Twitter

Latest Post

  • Advice For Building Team Culture In Modern Times
  • Is Your Business Infrastructure Up To Date And Is That Important?
  • How can a business degree help in succeeding businesses?
  • Digital Business Solutions You Might Not Be Taking Advantage Of
  • Field Service Reporting: Here is a Complete Guide to Field Service Reporting

NewsLetter

Featured

Advice For Building Team Culture In Modern Times

Is Your Business Infrastructure Up To Date And Is That Important?

Free Startup Meeting photo and picture

How can a business degree help in succeeding businesses?

Digital Business Solutions You Might Not Be Taking Advantage Of

Field Service Reporting: Here is a Complete Guide to Field Service Reporting

Free Woman in Red Sweater Leaning on White Table Stock Photo

5 Top MBA Concentrations to Consider

4 trending MBA specializations you can pursue

Ways To Prepare Cuttlefish

How Jobs To Be Done Helps Businesses and How It Works

Top Software Development Trends of 2023

About Us

Hello there! I’m Skyler a professional tech guy you can count on. I’m here to share with you my broad knowledge when it comes to tech stuff. I do some configuring and coding that might be helpful to you. Read More…

Latest Post

  • Advice For Building Team Culture In Modern Times
  • Is Your Business Infrastructure Up To Date And Is That Important?
  • How can a business degree help in succeeding businesses?
  • Digital Business Solutions You Might Not Be Taking Advantage Of
  • Field Service Reporting: Here is a Complete Guide to Field Service Reporting

NewsLetter

Connect With Us

  • Facebook
  • Instagram
  • LinkedIn
  • Twitter

Copyright © 2023 Your Own Linux