Copy Speed

I use Robocopy to copy backups from an internal drive to a removable
drive. I just do not understand why very large files (e.g., 15.8 GB)
copy slower than somewhat large files (e.g., 1.1 GB). When I say
"slower", I mean in terms of Bytes per second. The very large files
copy at a rate of 25 MB/sec while the somewhat large files copy at a
rate of 167 MB/sec, about 6.6 times faster. In both cases, the same
internal drive partition is being copied to the same removable drive, in
the same session.

Does anyone know of why this slowdown exists for very large files?

--
David E. Ross
http://www.rossde.com

Consider:
* Most state mandate that drivers have liability insurance.
* Employers are mandated to have worker's compensation insurance.
* If you live in a flood zone, flood insurance is mandatory.
* If your home has a mortgate, fire insurance is mandatory.

Why then is mandatory health insurance so bad??

En el artículo , David E. Ross
escribió:

Does anyone know of why this slowdown exists for very large files?

There is no slowdown. Windows is notorious for lying to you about the
speed at which it is copying.

--
(\_/)
(='.'=) systemd: the Linux version of Windows 10
(")_(")

David E. Ross wrote:
I use Robocopy to copy backups from an internal drive to a removable
drive. I just do not understand why very large files (e.g., 15.8 GB)
copy slower than somewhat large files (e.g., 1.1 GB). When I say
"slower", I mean in terms of Bytes per second. The very large files
copy at a rate of 25 MB/sec while the somewhat large files copy at a
rate of 167 MB/sec, about 6.6 times faster. In both cases, the same
internal drive partition is being copied to the same removable drive, in
the same session.

Does anyone know of why this slowdown exists for very large files?


Windows has a system write cache.

Whether it is used or not, depends on the settings for
the disk drive. For example, whether SATA hotplug is enabled
or not, influences the behavior. Or whether "optimize for
quick removal" is set.

Only a percentage of system RAM is used for write cache (whereas
the whole thing can be used for transparent read cache). By
observation, on my Test Machine, the write cache can go as
high as 5GB of content. Using Task Manager, you can watch
the RAM usage bloat up, as the write cache goes into action.
The RAM space used by the write cache is "booked", so it is
accounted for in Task Manager. The read cache on the other hand,
content is evicted on demand, so there is no reason to show
a graphical representation of its usage. The read cache design
is a purely opportunistic thing - you use the read cache if
some data is available, but if the read data is all evicted,
it's no big deal (just read the disk again instead).

If I were to copy from a faster disk to a slower disk on
my Test Machine, the fast disk output would end up in
the write cache, waiting for the slow disk to finish. Since
the 1.1GB file fits *entirely* in my write cache, I
get the impression of "tremendous speed". That's because
you're seeing the speed of a modern 167MB/sec source disk.

In the 15.8GB case, if I ran that on my setup, the first
5GB would be transferred (to cache) at 167MB/sec. But
the next 10.8GB, could only enter the cache, as quickly
as the USB2 enclosure can drain the cache (25MB/sec).

Now, after the copy operation is completed, the software
thinks the transfer is "done". Yet, on my Test Machine,
whatever is left in the write cache is still draining.
It can take at least 50 seconds for the write cache to
drain here. And, this is a dangerous design, in that if
the power goes off, I lose 50 seconds of "stuff" that the
system thinks is committed. I don't really know what blocks
that cache, whether there could be thousands of files sitting
in the cache, and what the impact would be if the power
goes off. The machine is on a UPS, but still...

If you managed to configure things such that a write cache
is not used, then the reported transfer rate will be
boringly consistent and "honest" (25MB/sec).

On machines with a small amount of RAM (say 4GB), the user
may not even be aware Windows has a write cache. I first
saw this behavior on my new computer. I was using FFMPEG at
the time, and I thought "oh, this is an FFMPEG deature".
Nope. It showed up while doing other things. I would never
have guessed Microsoft would implement a write cache, but
apparently, they did. And I don't know if I'm particularly
impressed - it still takes the usual amount of time for the
slow disk to finish, so in many scenarios, there is no "win"
by doing this. It just means an interfering disk activity could
hound the user (the 50 seconds of cache draining), when all
visible evidence is the system is supposed to be idle.

And when I speak of cache, I use that term to imply a hidden
reservoir. The "discipline" is actually FIFO (first in first out
queue). If the destination drive is faster than the source drive,
the queue occupancy is zero, because nothing can build up
since the drain works so well. When the destination is slower,
the FIFO accumulates the difference in speed, until the FIFO
is full, and then it can only fill, at the same rate it is
draining (25MB/sec).

Since the write cache competes with your copy of Microsoft Word,
the clever OS designers considered it would be a bad move,
if the write cache could occupy all of memory. Imagine your
frustration, if the write cache prevented MSWord from
starting. To prevent that, only a percentage of system RAM
is allowed to be used for a write cache.

Paul

On 3/15/2017 10:50 PM, Paul wrote:
David E. Ross wrote:
I use Robocopy to copy backups from an internal drive to a removable
drive. I just do not understand why very large files (e.g., 15.8 GB)
copy slower than somewhat large files (e.g., 1.1 GB). When I say
"slower", I mean in terms of Bytes per second. The very large files
copy at a rate of 25 MB/sec while the somewhat large files copy at a
rate of 167 MB/sec, about 6.6 times faster. In both cases, the same
internal drive partition is being copied to the same removable drive, in
the same session.

Does anyone know of why this slowdown exists for very large files?


Windows has a system write cache.

Whether it is used or not, depends on the settings for
the disk drive. For example, whether SATA hotplug is enabled
or not, influences the behavior. Or whether "optimize for
quick removal" is set.

Only a percentage of system RAM is used for write cache (whereas
the whole thing can be used for transparent read cache). By
observation, on my Test Machine, the write cache can go as
high as 5GB of content. Using Task Manager, you can watch
the RAM usage bloat up, as the write cache goes into action.
The RAM space used by the write cache is "booked", so it is
accounted for in Task Manager. The read cache on the other hand,
content is evicted on demand, so there is no reason to show
a graphical representation of its usage. The read cache design
is a purely opportunistic thing - you use the read cache if
some data is available, but if the read data is all evicted,
it's no big deal (just read the disk again instead).

If I were to copy from a faster disk to a slower disk on
my Test Machine, the fast disk output would end up in
the write cache, waiting for the slow disk to finish. Since
the 1.1GB file fits *entirely* in my write cache, I
get the impression of "tremendous speed". That's because
you're seeing the speed of a modern 167MB/sec source disk.

In the 15.8GB case, if I ran that on my setup, the first
5GB would be transferred (to cache) at 167MB/sec. But
the next 10.8GB, could only enter the cache, as quickly
as the USB2 enclosure can drain the cache (25MB/sec).

Now, after the copy operation is completed, the software
thinks the transfer is "done". Yet, on my Test Machine,
whatever is left in the write cache is still draining.
It can take at least 50 seconds for the write cache to
drain here. And, this is a dangerous design, in that if
the power goes off, I lose 50 seconds of "stuff" that the
system thinks is committed. I don't really know what blocks
that cache, whether there could be thousands of files sitting
in the cache, and what the impact would be if the power
goes off. The machine is on a UPS, but still...

If you managed to configure things such that a write cache
is not used, then the reported transfer rate will be
boringly consistent and "honest" (25MB/sec).

On machines with a small amount of RAM (say 4GB), the user
may not even be aware Windows has a write cache. I first
saw this behavior on my new computer. I was using FFMPEG at
the time, and I thought "oh, this is an FFMPEG deature".
Nope. It showed up while doing other things. I would never
have guessed Microsoft would implement a write cache, but
apparently, they did. And I don't know if I'm particularly
impressed - it still takes the usual amount of time for the
slow disk to finish, so in many scenarios, there is no "win"
by doing this. It just means an interfering disk activity could
hound the user (the 50 seconds of cache draining), when all
visible evidence is the system is supposed to be idle.

And when I speak of cache, I use that term to imply a hidden
reservoir. The "discipline" is actually FIFO (first in first out
queue). If the destination drive is faster than the source drive,
the queue occupancy is zero, because nothing can build up
since the drain works so well. When the destination is slower,
the FIFO accumulates the difference in speed, until the FIFO
is full, and then it can only fill, at the same rate it is
draining (25MB/sec).

Since the write cache competes with your copy of Microsoft Word,
the clever OS designers considered it would be a bad move,
if the write cache could occupy all of memory. Imagine your
frustration, if the write cache prevented MSWord from
starting. To prevent that, only a percentage of system RAM
is allowed to be used for a write cache.
Paul

Thanks for explaining that Paul, I never heard of that before, nice to
know.

well, its true, there are switches. they make a software that
controls the cache, that you can put on your desktop, I did have but I
never use it now.... It manipulates those switches of the ram cache
and the disk drive, and even can make a cache on the disc itself for
temporary use. So its just a matter of reset for smaller files.

But whats his name is right about windows being a liar.
You can find that stupidness in the path work of the explorer file
viewer, for one thing is certain, those file folders in those listings,
are all screwed up name wise, and so is the path to them also. And
that, is a real big trouble in today's cocky world.... It causes all
kinds of grief and breakdowns. But that's what you get in 666.
Same as, there is no such thing as Windows 7. It is really, Windows 6.1



On 3/15/2017 5:02 PM, David E. Ross wrote:
I use Robocopy to copy backups from an internal drive to a removable
drive. I just do not understand why very large files (e.g., 15.8 GB)
copy slower than somewhat large files (e.g., 1.1 GB). When I say
"slower", I mean in terms of Bytes per second. The very large files
copy at a rate of 25 MB/sec while the somewhat large files copy at a
rate of 167 MB/sec, about 6.6 times faster. In both cases, the same
internal drive partition is being copied to the same removable drive, in
the same session.

Does anyone know of why this slowdown exists for very large files?

En el artículo , David E. Ross
escribió:

I just do not understand why very large files (e.g., 15.8 GB)
copy slower

https://xkcd.com/612/

--
(\_/)
(='.'=) "Between two evils, I always pick
(")_(") the one I never tried before." - Mae West

Mike Tomlinson wrote:
En el artículo , David E. Ross
escribió:

I just do not understand why very large files (e.g., 15.8 GB)
copy slower

https://xkcd.com/612/


It's easier to figure out, with the graph mode.

http://media.askvg.com/articles/imag...ox_Details.png

Paul