Large file copy behavior question

I user Acronis to produce partition backups. They wind up
on an otherwise unused HD partition and I then copy them
to an external USB drive. I have a full backup scheduled
for once a week followed by six incrementals. The full
backup is about 450GB. I've noticed something puzzling
when copying to the external drive (when the system is
otherwise idle). The copy pops open that nifty little
window that plots the progress of the copy and shows the
instantaneous transfer rate. When the copy starts up, it
proceeds at around 110 Mb/s but then often, not always,
slows down to about half that rate. Sometimes it proceeds
to the end at that speed, other times the speed will jump
back up to 100+ and stay there until finished. Sometimes
it will start at 100+ and then very gradually slow down
until it's 30-40 MB/s for the duration of the operation.
The copy process takes more than an hour. Why does the
speed vary like this? I've watched Task Manager for clues
but haven't spotted any. The copy procedure is about the
only thing consuming any resources and not many at that -
CPU usage is about 4% though the internal HDD is max'd
out, even when the copy is proceeding slowly.

In article -
september.org, says...
it
proceeds at around 110 Mb/s


Mb/s should be MB/s everywhere in my original post

Jason wrote:
I user Acronis to produce partition backups. They wind up
on an otherwise unused HD partition and I then copy them
to an external USB drive. I have a full backup scheduled
for once a week followed by six incrementals. The full
backup is about 450GB. I've noticed something puzzling
when copying to the external drive (when the system is
otherwise idle). The copy pops open that nifty little
window that plots the progress of the copy and shows the
instantaneous transfer rate. When the copy starts up, it
proceeds at around 110 Mb/s but then often, not always,
slows down to about half that rate.

You're probably filling up a write cache on the drive with the large copy
and it can't sustain the speed when writing to the actual disk.

50-60MB/s is about the maximum speed for a USB hard disk.

Jason wrote:

Why does the speed vary like this?

I'm assuming at those sizes, that the external USB disk isn't an SSD,
but maybe it's a hybrid SSHD? (e.g. 8GB of flash + 1TB of rust)


p.s. you've used a mixture of Mb and MB which makes talking about your
actual numbers rather awkward.

Jason wrote:
I user Acronis to produce partition backups. They wind up
on an otherwise unused HD partition and I then copy them
to an external USB drive. I have a full backup scheduled
for once a week followed by six incrementals. The full
backup is about 450GB. I've noticed something puzzling
when copying to the external drive (when the system is
otherwise idle). The copy pops open that nifty little
window that plots the progress of the copy and shows the
instantaneous transfer rate. When the copy starts up, it
proceeds at around 110 Mb/s but then often, not always,
slows down to about half that rate. Sometimes it proceeds
to the end at that speed, other times the speed will jump
back up to 100+ and stay there until finished. Sometimes
it will start at 100+ and then very gradually slow down
until it's 30-40 MB/s for the duration of the operation.
The copy process takes more than an hour. Why does the
speed vary like this? I've watched Task Manager for clues
but haven't spotted any. The copy procedure is about the
only thing consuming any resources and not many at that -
CPU usage is about 4% though the internal HDD is max'd
out, even when the copy is proceeding slowly.


Isn't Windows marvelous ?

First, you have to look at your ecosystem.

What is the transfer rate of the source drive ?

What is the transfer rate of the destination drive ?
Is the destination drive or device, actually USB2, or
is a USB3 port and USB3 enclosure controller being used ?

If you're transferring 450GB, you'd want a USB3 path.
USB2 would take forever to transfer that much.

When the source is faster than the dest, the OS uses the
"system write cache". The system write cache is "charged"
against system memory. It uses memory so that programs
cannot use the memory at the same time. The cache may
occupy about 12.5% or so of total system memory. On
a 16GB system, perhaps 2GB functions as a System Write Cache.
Once writes are finished (the LED stops flashing), that
2GB of RAM returns to the system pool.

If you have a good characterization of your "resting RAM usage",
you can spot the usage (and draining) of the System Write
Cache in Task Manager, while the transfer runs.

When the transfer first starts, the source drive runs at
its rate. If the destination is too slow, the System
Write Cache functions as a FIFO queue. The queue fills
up at source rate. The initial part of the transfer
appears to be happening at "miraculous speed", when
all it's doing, is deceiving you by filling a FIFO queue.

Before the destination device can be unplugged, the FIFO
queue has to be empty. If you were to use Safely Remove,
it will need to drain the FIFO queue first. If the queue
held 2GB of data, the destination cable limits transfer
rate to 30MB/sec, it will take a bit more than a minute
for Safely Remove to really be ready. The FIFO queue drains
without prompting, but if you're in a rush to unplug,
or if you're in a rush to do a Shutdown, you'll be waiting
up to a minute for the remains of the queued write to
finish.

Source_rate --- FIFO Queue -- Destination_rate

What other processes are running on the computer ?
There is Windows Defender, with realtime scanning enabled.
To make life interesting, it can be scanning. If you
were running 7ZIP and dumping a large number of small
files, into a .7Z stored on the external drive, then
Windows Defender can slow down the source rate by
a factor of 2x or 3x. Sometimes, you will see a lot
of strange looking dips in the transfer curve, and
Windows Defender is your friend on that.

Another note on Source Rate. Say you have an .mrimg,
it's relatively small, and you run Verify on it.
Every bit of the .mrimg would be read and checksummed
as part of Verify. The System Read Cache (which doesn't
officially use RAM and gives up RAM instantly when it is
needed) can cache everything that is read. Say your
16GB machine has about 14GB free. Then if the file you
just read from end-to-end was 10GB, the entire file
is now in RAM. This behavior, of keeping a System
Read Cache, is available on all modern OSes, and is
considered a free lunch.

Now, consider what happens when you transfer the same
file. We want to transfer our 10GB file. Initially,
instead of reading at Source_rate, File Explorer gets
the 10GB file from the System Read Cache. It transfers
the System Read Cache to the System Write Cache, at
maybe 1-2GB/sec. The transfer curve then has a *huge*
lump at the beginning. In a perfect world, the System
Read Cache would be used for the entire transfer, but
it doesn't have to be, and then if the Source drive is
consulted again, the Source_rate will apply.

It's a lot like a Slinky :-) With lots of pot stirring
and shenanigans thrown in. And in the end, maybe the entire
transfer isn't any faster, but the antics will
keep you amused for hours.

*******

So what could you do about it ? You could disable
write caching. That might make the transfer more
synchronous and predictable. If dumping to USB2,
then the transfer rate should remain a steady 35MB/sec
if UASP protocol is used. And then the transfer
curve would be flat. Especially if you disabled
Windows Defender to put it out of the picture,
you disabled SearchIndexer on the destination
drive, and so on.

In some cases, I try to look away during a
transfer, because I really don't want to "learn
any more about it". I can tell you though, I've
developed a habit of disabling Windows Defender,
when it interferes with performance. For example,
if I want to run Hashdeep and generate checksums
for all files on the machine, I disable Windows
Defender real-time scanning before I begin, because
of the penalty it causes. Maybe my hashdeep runs
3x faster with WD disabled.

There is one other thing on the machine which
can create a load, and that's Superfetch service.
I don't know what's up with that, but sometimes
it seemingly goes nuts, and I have to bring
up the Services panel and stop it. If it has
a real purpose, I'm not aware of any benefit on
my test system. Since my Test System seems to have
a problem in Windows 10, telling the difference
between an SSD and a HDD, it could be that the
status of the drive, is causing Superfetch to do
stuff it shouldn't be doing (for an SSD). SSDs don't
need file positioning, since the "seek time is zero".

So when in a demanding situation on Windows 10, watch:

Windows Defender (MsMpEng)
SearchIndexer (three processes in Task Manager, try Services panel)
SuperFetch service (hides in a busy SVCHOST, must use ProcExp to ID
or tasklist /svc, use Services panel to disable)

Windows Defender has a GUI, as well as a PowerShell command.
SearchIndexer uses the Services panel, but "won't take
no for an answer". That's a very annoying service.
SuperFetch is well behaved, and the Services panel
can switch it off successfully, without a fuss.
SearchIndexer ("Windows Search" service) is a
whack-a-mole thing you can stop twenty times, and
it'll still keep popping up. Even if you set the
Recovery Policy on SearchIndexer to "Do nothing on
failure", it'll still keep restarting.

That's potentially three controls to play with.

You can set a device to "Optimize for Fast Removal" if
you want to disable write cache behavior. Check Device
Manager, and Policies tab for a drive, to see what
options are offered. The Uwe Sieber site has more
information on USB devices in particular. Note that
in Windows, internal drives can have a "removable"
status due to SATA hotswap capability, and that can
modify their Policy tab behaviors.

"Removable or what?"

https://www.uwe-sieber.de/usbstick_e.html

I wouldn't have written all of the above, except I continue
to have a suspicion the System Write Cache doesn't work
properly and stutters on occasion. No data is "hurt", but
the transfer rate just doesn't seem to be right. Even considering
all the sources of interference and dealing with them.

Paul

Paul wrote:

Before the destination device can be unplugged, the FIFO
queue has to be empty. If you were to use Safely Remove,
it will need to drain the FIFO queue first.

I'm getting more and more frustrated with "safely remove" it mostly
seems to claim the drive is still in use, this happens with USB mass
storage, USB attached SCSI (UASP) and eSATA drives

Various things sometime help, but none seems to help in all cases.

Closing all explorer windows
Killing all explorer processes
Closing all apps and logging out
Looking for open handles with procmon

It's as if they *want* us to just yank the cable ... grr!

In article , Chris
wrote:

50-60MB/s is about the maximum speed for a USB hard disk.

i normally get double that.

On 23/04/2018 09:58:29, Andy Burns wrote:
Paul wrote:

Before the destination device can be unplugged, the FIFO
queue has to be empty. If you were to use Safely Remove,
it will need to drain the FIFO queue first.

I'm getting more and more frustrated with "safely remove" it mostly seems to
claim the drive is still in use, this happens with USB mass storage, USB
attached SCSI (UASP) and eSATA drives

Various things sometime help, but none seems to help in all cases.

Closing all explorer windows
Killing all explorer processes
Closing all apps and logging out
Looking for open handles with procmon

It's as if they *want* us to just yank the cable ... grr!

Having the same problem here about 90% of the time.
I shut down the pc at the end of the day so I wait until then and pull
the usb cable. If I need the device disconnecting before that I shut
down, pull the cable and start up again as it only takes seconds
booting up again with an ssd drive. Slightly annoying but I can put up
with that.

--
mick

Andy Burns wrote:
Paul wrote:

Before the destination device can be unplugged, the FIFO
queue has to be empty. If you were to use Safely Remove,
it will need to drain the FIFO queue first.

I'm getting more and more frustrated with "safely remove" it mostly
seems to claim the drive is still in use, this happens with USB mass
storage, USB attached SCSI (UASP) and eSATA drives

Various things sometime help, but none seems to help in all cases.

Closing all explorer windows
Killing all explorer processes
Closing all apps and logging out
Looking for open handles with procmon

It's as if they *want* us to just yank the cable ... grr!

Closing Explorer windows (even when it doesn't seem possible
the windows in question have a "handle" on the disk), seems
to work here.

In the most pestilent cases, a tool you've been using, has
used the TXF transactional NTFS feature, and (something) continues
to keep that open. The only thing I've been able to figure
out for that case, is go to Disk Management, click the
box on the left of that row of Disk Management, and select
Offline. Once the drive is Offline, Safely Remove works.
There don't seem to be a lot of victims for that one.
And the feature in question is now "deprecated" in Windows 10
for some reason. It makes file operations on NTFS "atomic",
implying a single bit flip somewhere, commits a file (or
doesn't commit it). Whatever TxF is, it has its own
journal files (that's how I detected it, was some signs
of journal fluff).

https://en.wikipedia.org/wiki/Transactional_NTFS

The next time you plug that particular drive in,
you have to go to Disk Management *again* and select
"Online" to allow it to mount.

Paul

Paul wrote:

go to Disk Management, click the
box on the left of that row of Disk Management, and select
Offline. Once the drive is Offline, Safely Remove works.

I've used that a time or two, but am not really clear if it's any
"nicer" to whatever is clinging to the disk than unplugging ...

Paul wrote:

The next time you plug that particular drive in,
you have to go to Disk Management*again* and select
"Online" to allow it to mount.

I've been tinkering with powershell's set-disk cmdlet to flip the
isOffline and isReadOnly bits on my 4TB backup drive that's connected
via my docking station ... seems to eject without question, and may
offer some protection against accidental (deliberate?) deletion.

Jason wrote:
I user Acronis to produce partition backups. They wind up
on an otherwise unused HD partition and I then copy them
to an external USB drive. I have a full backup scheduled
for once a week followed by six incrementals. The full
backup is about 450GB. I've noticed something puzzling
when copying to the external drive (when the system is
otherwise idle). The copy pops open that nifty little
window that plots the progress of the copy and shows the
instantaneous transfer rate. When the copy starts up, it
proceeds at around 110 Mb/s but then often, not always,
slows down to about half that rate. Sometimes it proceeds
to the end at that speed, other times the speed will jump
back up to 100+ and stay there until finished. Sometimes
it will start at 100+ and then very gradually slow down
until it's 30-40 MB/s for the duration of the operation.
The copy process takes more than an hour. Why does the
speed vary like this? I've watched Task Manager for clues
but haven't spotted any. The copy procedure is about the
only thing consuming any resources and not many at that -
CPU usage is about 4% though the internal HDD is max'd
out, even when the copy is proceeding slowly.


Make a batch so your work time is not impacted by copy time:
1. Check if external drive is available to use as destination. If not use
internal.
(Incase you forgot to attach external)
2. Make the image.
3. If desired set Acronis to shut down PC after image is completed.

Batch commands can be added to control full vs. incremental images by Day
of Week and also saving a weeks images if room on external drive.

Run the batch manually from icon or automatically from Time Scheduler.
--
Zaidy036

On 04/22/2018 08:23 PM, Jason wrote:
In article -
september.org, says...
it
proceeds at around 110 Mb/s


Mb/s should be MB/s everywhere in my original post

A lot of people mix up big B (bytes) with little b (bits).

110 MB/s = 880 Mb/s

Wolf K wrote:
On 2018-04-23 07:06, Paul wrote:
Closing Explorer windows (even when it doesn't seem possible
the windows in question have a "handle" on the disk), seems
to work here.

Explorer checks all drives on boot, including external ones. The tree
view shows all powered drives. So closing the Explorer window(s) and
using the command-line should make for a faster Copy.

Or????


If you use Safely Remove, to remove a USB device, sometimes
you get a complaint the "volume is busy".

The obvious answer, is some program running on your machine,
has a file open on the volume in question, perhaps about
to write it.

Now, if you use Process Explorer or Handle.exe, you can check
to see exactly which files are open. And then close the programs
that have the files in question open.

OK, what happens if Safely Remove *still* won't remove the
drive ? It's still busy it seems.

One thing you can do, is close the File Explorer windows, as
sometimes, even though the File Explorer window isn't displaying
the volume contents in question, it still seems to have a handle
open on the drive.

Even after you've done that, there can still be cases where it
won't release so you can unplug it. One source of contention,
is the NTFS TxF feature, is keeping TxF files open on the
volume. Using the Disk Management "Offline" command seems
to help in that case.

Shutting down the machine should always release the volume,
but you might see a slow shutdown process, if the "culprit"
is gasping for breath at shutdown :-) There are some things
which don't go gracefully.

I had one of these the other day on this machine, and I
had to power cycle to regain control. I restored from backup,
because I love to do stuff like that... Just to make sure
whatever happened, didn't leave any after-effects. The
volume wasn't actually damaged, but since I had the backup,
I used it.

Paul

On 23/04/2018 14:50:17, Paul wrote:
Wolf K wrote:
On 2018-04-23 07:06, Paul wrote:
Closing Explorer windows (even when it doesn't seem possible
the windows in question have a "handle" on the disk), seems
to work here.

Explorer checks all drives on boot, including external ones. The tree view
shows all powered drives. So closing the Explorer window(s) and using the
command-line should make for a faster Copy.

Or????


If you use Safely Remove, to remove a USB device, sometimes
you get a complaint the "volume is busy".

The obvious answer, is some program running on your machine,
has a file open on the volume in question, perhaps about
to write it.

Now, if you use Process Explorer or Handle.exe, you can check
to see exactly which files are open. And then close the programs
that have the files in question open.

OK, what happens if Safely Remove *still* won't remove the
drive ? It's still busy it seems.

One thing you can do, is close the File Explorer windows, as
sometimes, even though the File Explorer window isn't displaying
the volume contents in question, it still seems to have a handle
open on the drive.

Even after you've done that, there can still be cases where it
won't release so you can unplug it. One source of contention,
is the NTFS TxF feature, is keeping TxF files open on the
volume. Using the Disk Management "Offline" command seems
to help in that case.

Shutting down the machine should always release the volume,
but you might see a slow shutdown process, if the "culprit"
is gasping for breath at shutdown :-) There are some things
which don't go gracefully.

I had one of these the other day on this machine, and I
had to power cycle to regain control. I restored from backup,
because I love to do stuff like that... Just to make sure
whatever happened, didn't leave any after-effects. The
volume wasn't actually damaged, but since I had the backup,
I used it.

Paul

I do not use File Explorer and nothing is using the external disk
AFAIK.
I have watched the external disk light for ages which is steady, so
nothing is writing to it.
The external disk is only used for syncing to, using Allway Sync, the
only program to be opened and closed in the session. Sometimes the
drive can be ejected other times it is held onto saying it is in use.

--
mick