Your Own Linux

Tech Blog

  • Home
  • About
  • Blog
  • Technology
  • Business
  • Contact Us

How To Speed Up File Transfers Using Rsync With GNU Parallel

August 31, 2022 by Skyler Leave a Comment

When you’re uploading or downloading large files to or from a remote server, your connection speed will be limited by the maximum throughput of your Internet link. The rsync command-line utility can help you make the most of your available bandwidth by transferring files to and from the remote host in chunks as small as you specify, which means that Rsync can transfer files much faster than if it transferred the entire file before moving on to the next one. In this article, I’ll explain how to ,speed up file transfers using rsync with gnu parallel, and show you how to use them together to speed up file transfers on rsync multiple threads.

Backup your files first

Before you do anything, it’s always a good idea to backup your files. That way, if something goes wrong, you have a copy of the original data. There are many ways to backup files, but we recommend using rsync. rsync is a fast and efficient way to transfer files and can be used in conjunction with GNU Parallel to speed up the process. Here’s how rsync -rltp -nxt Ipsum * remote:/backup/incoming-local-directory/ gnu parallel –strip=2 ‘rsync -to -nxt Ipsum *.txt remote:/backup/incoming-local-directory/’ ::: ipsum1.txt,ipsum2.txt

Getting Started

rsync is a great tool for copying files between servers. It’s fast and efficient, but it can be even faster with the help of GNU Parallel. By using both tools together, you can speed up file transfers and get your work done more quickly. Here’s how to do it ;

– First, install GNU Parallel on both machines: sudo apt-get install parallel

– To do this, first copy the original file with rsync (e.g., rsync -av /source/path/ /destination/path). Then use parallel (e.g., parallel -j4 ‘rsync -av –delete-after /source/path/* /destination/path’) to copy all of the updated versions at once.

Create two lists of files – one for local source and one for remote destination echo file1 > list_local $ echo file2 >> list_local echo file3 >> list_local echo file4 >> list_remote # Copy the local files to the remote server:

rsync -av –delete-after –progress ./list_local ./list_remote/

Basic Example of rsync + gnu parallel

If you have a lot of files to transfer, using rsync in combination with GNU parallel can speed up the process. Here’s a basic example of how it works rsync -av –progress /source/directory /destination/directory

gnu parallel -j 4 –max-lines=1000000 rsync -av –progress /source/directory /destination/directory

Conclusions

rsync is a great tool for quickly copying files between two locations. However, it can be slow when copying large numbers of files. GNU Parallel is a great tool for speeding up file transfers by using multiple processors. When used together, these two tools can significantly speed up the process of copying files.

Filed Under: Technology

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

About Us

Hello there! I’m Skyler a professional tech guy you can count on. I’m here to share with you my broad knowledge when it comes to tech stuff. I do some configuring and coding that might be helpful to you. Read More…

Connect With Us

  • Facebook
  • Instagram
  • LinkedIn
  • Twitter

Latest Post

  • How to Use Stock Photos for Your Business
  • 6 Green Marketing Strategies Businesses Should Adopt
  • Tips to Help You Play Solitaire Like a Pro and Win
  • Best Coding Homework Help Websites
  • The Best Nonogram Puzzles You Can Play On Your Smartphone Today

NewsLetter

Featured

Free Books Smartphone photo and picture

How to Use Stock Photos for Your Business

Free Working in a Group Stock Photo

6 Green Marketing Strategies Businesses Should Adopt

Tips to Help You Play Solitaire Like a Pro and Win

Best Coding Homework Help Websites

The Best Nonogram Puzzles You Can Play On Your Smartphone Today

The Best Ways To Use Tech As A Delivery Driver

Advice For Building Team Culture In Modern Times

Is Your Business Infrastructure Up To Date And Is That Important?

Free Startup Meeting photo and picture

How can a business degree help in succeeding businesses?

Digital Business Solutions You Might Not Be Taking Advantage Of

About Us

Hello there! I’m Skyler a professional tech guy you can count on. I’m here to share with you my broad knowledge when it comes to tech stuff. I do some configuring and coding that might be helpful to you. Read More…

Latest Post

  • How to Use Stock Photos for Your Business
  • 6 Green Marketing Strategies Businesses Should Adopt
  • Tips to Help You Play Solitaire Like a Pro and Win
  • Best Coding Homework Help Websites
  • The Best Nonogram Puzzles You Can Play On Your Smartphone Today

NewsLetter

Connect With Us

  • Facebook
  • Instagram
  • LinkedIn
  • Twitter

Copyright © 2023 Your Own Linux