If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Rate Thread | Display Modes |
#1
|
|||
|
|||
Copy Speed
I use Robocopy to copy backups from an internal drive to a removable
drive. I just do not understand why very large files (e.g., 15.8 GB) copy slower than somewhat large files (e.g., 1.1 GB). When I say "slower", I mean in terms of Bytes per second. The very large files copy at a rate of 25 MB/sec while the somewhat large files copy at a rate of 167 MB/sec, about 6.6 times faster. In both cases, the same internal drive partition is being copied to the same removable drive, in the same session. Does anyone know of why this slowdown exists for very large files? -- David E. Ross http://www.rossde.com Consider: * Most state mandate that drivers have liability insurance. * Employers are mandated to have worker's compensation insurance. * If you live in a flood zone, flood insurance is mandatory. * If your home has a mortgate, fire insurance is mandatory. Why then is mandatory health insurance so bad?? |
Ads |
#2
|
|||
|
|||
Copy Speed
En el artículo , David E. Ross
escribió: Does anyone know of why this slowdown exists for very large files? There is no slowdown. Windows is notorious for lying to you about the speed at which it is copying. -- (\_/) (='.'=) systemd: the Linux version of Windows 10 (")_(") |
#3
|
|||
|
|||
Copy Speed
David E. Ross wrote:
I use Robocopy to copy backups from an internal drive to a removable drive. I just do not understand why very large files (e.g., 15.8 GB) copy slower than somewhat large files (e.g., 1.1 GB). When I say "slower", I mean in terms of Bytes per second. The very large files copy at a rate of 25 MB/sec while the somewhat large files copy at a rate of 167 MB/sec, about 6.6 times faster. In both cases, the same internal drive partition is being copied to the same removable drive, in the same session. Does anyone know of why this slowdown exists for very large files? Windows has a system write cache. Whether it is used or not, depends on the settings for the disk drive. For example, whether SATA hotplug is enabled or not, influences the behavior. Or whether "optimize for quick removal" is set. Only a percentage of system RAM is used for write cache (whereas the whole thing can be used for transparent read cache). By observation, on my Test Machine, the write cache can go as high as 5GB of content. Using Task Manager, you can watch the RAM usage bloat up, as the write cache goes into action. The RAM space used by the write cache is "booked", so it is accounted for in Task Manager. The read cache on the other hand, content is evicted on demand, so there is no reason to show a graphical representation of its usage. The read cache design is a purely opportunistic thing - you use the read cache if some data is available, but if the read data is all evicted, it's no big deal (just read the disk again instead). If I were to copy from a faster disk to a slower disk on my Test Machine, the fast disk output would end up in the write cache, waiting for the slow disk to finish. Since the 1.1GB file fits *entirely* in my write cache, I get the impression of "tremendous speed". That's because you're seeing the speed of a modern 167MB/sec source disk. In the 15.8GB case, if I ran that on my setup, the first 5GB would be transferred (to cache) at 167MB/sec. But the next 10.8GB, could only enter the cache, as quickly as the USB2 enclosure can drain the cache (25MB/sec). Now, after the copy operation is completed, the software thinks the transfer is "done". Yet, on my Test Machine, whatever is left in the write cache is still draining. It can take at least 50 seconds for the write cache to drain here. And, this is a dangerous design, in that if the power goes off, I lose 50 seconds of "stuff" that the system thinks is committed. I don't really know what blocks that cache, whether there could be thousands of files sitting in the cache, and what the impact would be if the power goes off. The machine is on a UPS, but still... If you managed to configure things such that a write cache is not used, then the reported transfer rate will be boringly consistent and "honest" (25MB/sec). On machines with a small amount of RAM (say 4GB), the user may not even be aware Windows has a write cache. I first saw this behavior on my new computer. I was using FFMPEG at the time, and I thought "oh, this is an FFMPEG deature". Nope. It showed up while doing other things. I would never have guessed Microsoft would implement a write cache, but apparently, they did. And I don't know if I'm particularly impressed - it still takes the usual amount of time for the slow disk to finish, so in many scenarios, there is no "win" by doing this. It just means an interfering disk activity could hound the user (the 50 seconds of cache draining), when all visible evidence is the system is supposed to be idle. And when I speak of cache, I use that term to imply a hidden reservoir. The "discipline" is actually FIFO (first in first out queue). If the destination drive is faster than the source drive, the queue occupancy is zero, because nothing can build up since the drain works so well. When the destination is slower, the FIFO accumulates the difference in speed, until the FIFO is full, and then it can only fill, at the same rate it is draining (25MB/sec). Since the write cache competes with your copy of Microsoft Word, the clever OS designers considered it would be a bad move, if the write cache could occupy all of memory. Imagine your frustration, if the write cache prevented MSWord from starting. To prevent that, only a percentage of system RAM is allowed to be used for a write cache. Paul |
#4
|
|||
|
|||
Copy Speed
On 3/15/2017 10:50 PM, Paul wrote:
David E. Ross wrote: I use Robocopy to copy backups from an internal drive to a removable drive. I just do not understand why very large files (e.g., 15.8 GB) copy slower than somewhat large files (e.g., 1.1 GB). When I say "slower", I mean in terms of Bytes per second. The very large files copy at a rate of 25 MB/sec while the somewhat large files copy at a rate of 167 MB/sec, about 6.6 times faster. In both cases, the same internal drive partition is being copied to the same removable drive, in the same session. Does anyone know of why this slowdown exists for very large files? Windows has a system write cache. Whether it is used or not, depends on the settings for the disk drive. For example, whether SATA hotplug is enabled or not, influences the behavior. Or whether "optimize for quick removal" is set. Only a percentage of system RAM is used for write cache (whereas the whole thing can be used for transparent read cache). By observation, on my Test Machine, the write cache can go as high as 5GB of content. Using Task Manager, you can watch the RAM usage bloat up, as the write cache goes into action. The RAM space used by the write cache is "booked", so it is accounted for in Task Manager. The read cache on the other hand, content is evicted on demand, so there is no reason to show a graphical representation of its usage. The read cache design is a purely opportunistic thing - you use the read cache if some data is available, but if the read data is all evicted, it's no big deal (just read the disk again instead). If I were to copy from a faster disk to a slower disk on my Test Machine, the fast disk output would end up in the write cache, waiting for the slow disk to finish. Since the 1.1GB file fits *entirely* in my write cache, I get the impression of "tremendous speed". That's because you're seeing the speed of a modern 167MB/sec source disk. In the 15.8GB case, if I ran that on my setup, the first 5GB would be transferred (to cache) at 167MB/sec. But the next 10.8GB, could only enter the cache, as quickly as the USB2 enclosure can drain the cache (25MB/sec). Now, after the copy operation is completed, the software thinks the transfer is "done". Yet, on my Test Machine, whatever is left in the write cache is still draining. It can take at least 50 seconds for the write cache to drain here. And, this is a dangerous design, in that if the power goes off, I lose 50 seconds of "stuff" that the system thinks is committed. I don't really know what blocks that cache, whether there could be thousands of files sitting in the cache, and what the impact would be if the power goes off. The machine is on a UPS, but still... If you managed to configure things such that a write cache is not used, then the reported transfer rate will be boringly consistent and "honest" (25MB/sec). On machines with a small amount of RAM (say 4GB), the user may not even be aware Windows has a write cache. I first saw this behavior on my new computer. I was using FFMPEG at the time, and I thought "oh, this is an FFMPEG deature". Nope. It showed up while doing other things. I would never have guessed Microsoft would implement a write cache, but apparently, they did. And I don't know if I'm particularly impressed - it still takes the usual amount of time for the slow disk to finish, so in many scenarios, there is no "win" by doing this. It just means an interfering disk activity could hound the user (the 50 seconds of cache draining), when all visible evidence is the system is supposed to be idle. And when I speak of cache, I use that term to imply a hidden reservoir. The "discipline" is actually FIFO (first in first out queue). If the destination drive is faster than the source drive, the queue occupancy is zero, because nothing can build up since the drain works so well. When the destination is slower, the FIFO accumulates the difference in speed, until the FIFO is full, and then it can only fill, at the same rate it is draining (25MB/sec). Since the write cache competes with your copy of Microsoft Word, the clever OS designers considered it would be a bad move, if the write cache could occupy all of memory. Imagine your frustration, if the write cache prevented MSWord from starting. To prevent that, only a percentage of system RAM is allowed to be used for a write cache. Paul Thanks for explaining that Paul, I never heard of that before, nice to know. |
#5
|
|||
|
|||
Copy Speed
well, its true, there are switches. they make a software that
controls the cache, that you can put on your desktop, I did have but I never use it now.... It manipulates those switches of the ram cache and the disk drive, and even can make a cache on the disc itself for temporary use. So its just a matter of reset for smaller files. But whats his name is right about windows being a liar. You can find that stupidness in the path work of the explorer file viewer, for one thing is certain, those file folders in those listings, are all screwed up name wise, and so is the path to them also. And that, is a real big trouble in today's cocky world.... It causes all kinds of grief and breakdowns. But that's what you get in 666. Same as, there is no such thing as Windows 7. It is really, Windows 6.1 On 3/15/2017 5:02 PM, David E. Ross wrote: I use Robocopy to copy backups from an internal drive to a removable drive. I just do not understand why very large files (e.g., 15.8 GB) copy slower than somewhat large files (e.g., 1.1 GB). When I say "slower", I mean in terms of Bytes per second. The very large files copy at a rate of 25 MB/sec while the somewhat large files copy at a rate of 167 MB/sec, about 6.6 times faster. In both cases, the same internal drive partition is being copied to the same removable drive, in the same session. Does anyone know of why this slowdown exists for very large files? |
#6
|
|||
|
|||
Copy Speed
En el artículo , David E. Ross
escribió: I just do not understand why very large files (e.g., 15.8 GB) copy slower https://xkcd.com/612/ -- (\_/) (='.'=) "Between two evils, I always pick (")_(") the one I never tried before." - Mae West |
#7
|
|||
|
|||
Copy Speed
Mike Tomlinson wrote:
En el artículo , David E. Ross escribió: I just do not understand why very large files (e.g., 15.8 GB) copy slower https://xkcd.com/612/ It's easier to figure out, with the graph mode. http://media.askvg.com/articles/imag...ox_Details.png Paul |
Thread Tools | |
Display Modes | Rate This Thread |
|
|