英文:
CPU Processor vs number of threads
问题
当前情况：我使用进程构建器通过Java代码执行Rsync命令，从一台Linux机器传输一些大文件结构到另一台机器。
执行此Java代码（传输文件）的机器的处理器是八核的。
我的观察：当我创建一个任务调度程序，以并行执行8个线程时，我可以获得最大的传输速度。（这相当令人信服，因为我正在使用八核处理器）
我想要的情景：现在我想要使用32核处理器的机器来传输文件。我使用32个并行线程（固定大小为32的线程池）。
我提出这个问题是因为我没有32核的机器，无法进行测试。再添加一个复杂因素，有4台机器并行运行，总共有128个线程。我正在传输数据的机器只有一台。
问题：在传输速度方面，我会获得性能提升吗？是否有考虑目标机器的因素会限制我的传输速度？
英文:
Current Scenario : I use process builder to execute Rsync command through Java code to transfer some big file structures from one Linux machine to another.
The processor of the machine which executes this Java code(transfer files) is Octa core.
My observation : When I create a task scheduler which executes 8 threads in parallel, I get maximum speed for transfer. (quite convincing, as I am using Octa core processor)
Scenario I want : Now I want to use 32 core processor machine to transfer files. And I use 32 threads in parallel(thread pool of fix size 32)
I am asking this question because I donot have 32 core machine and cannot perform test.Adding one more complication, there are 4 machines in parallel which makes total threads 128.The machine where I am transferring data is only one.
Question : Will I get improved performance in terms of transfer speed? Is there any factor considering target machine which will limit my transfer speed?
答案1
得分: 0
"When I create a task scheduler which executes 8 threads in parallel, I get maximum speed for transfer...Now I want to use 32 core processor machine to transfer files...Will I get improved performance in terms of transfer speed? Is there any factor considering target machine which will limit my transfer speed?
Improved performance: the main bottle neck is the network bandwidth, not the amount of compute power. However, if rsync is using compression then yes, using more compute (more parallel threads) might help. And in your case, more CPU gave more throughput so let's assume that will happen
Other factors:
- the bandwidth of the sending side
- the bandwidth for the DC that you are receiving at
- if you are really hammering it then memory use might be an issue, see https://stackoverflow.com/questions/10105203/how-can-i-limit-the-cache-used-by-copying-so-there-is-still-memory-available-for/20817772#20817772"
英文:
> "When I create a task scheduler which executes 8 threads in parallel, I get maximum speed for transfer...Now I want to use 32 core processor machine to transfer files...Will I get improved performance in terms of transfer speed? Is there any factor considering target machine which will limit my transfer speed?
Improved performance: the main bottle neck is the network bandwidth, not the amount of compute power. However, if rsync is using compression then yes, using more compute (more parallel threads) might help. And in your case, more CPU gave more throughput so let's assume that will happen
Other factors:
- the bandwidth of the sending side
- the bandwidth for the DC that you are receiving at
- if you are really hammering it then memory use might be an issue, see https://stackoverflow.com/questions/10105203/how-can-i-limit-the-cache-used-by-copying-so-there-is-still-memory-available-for/20817772#20817772
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论