Large file copy behavior question

mick wrote:
On 23/04/2018 14:50:17, Paul wrote:
Wolf K wrote:
On 2018-04-23 07:06, Paul wrote:
Closing Explorer windows (even when it doesn't seem possible
the windows in question have a "handle" on the disk), seems
to work here.

Explorer checks all drives on boot, including external ones. The tree
view shows all powered drives. So closing the Explorer window(s) and
using the command-line should make for a faster Copy.

Or????


If you use Safely Remove, to remove a USB device, sometimes
you get a complaint the "volume is busy".

The obvious answer, is some program running on your machine,
has a file open on the volume in question, perhaps about
to write it.

Now, if you use Process Explorer or Handle.exe, you can check
to see exactly which files are open. And then close the programs
that have the files in question open.

OK, what happens if Safely Remove *still* won't remove the
drive ? It's still busy it seems.

One thing you can do, is close the File Explorer windows, as
sometimes, even though the File Explorer window isn't displaying
the volume contents in question, it still seems to have a handle
open on the drive.

Even after you've done that, there can still be cases where it
won't release so you can unplug it. One source of contention,
is the NTFS TxF feature, is keeping TxF files open on the
volume. Using the Disk Management "Offline" command seems
to help in that case.

Shutting down the machine should always release the volume,
but you might see a slow shutdown process, if the "culprit"
is gasping for breath at shutdown :-) There are some things
which don't go gracefully.

I had one of these the other day on this machine, and I
had to power cycle to regain control. I restored from backup,
because I love to do stuff like that... Just to make sure
whatever happened, didn't leave any after-effects. The
volume wasn't actually damaged, but since I had the backup,
I used it.

Paul

I do not use File Explorer and nothing is using the external disk AFAIK.
I have watched the external disk light for ages which is steady, so
nothing is writing to it.
The external disk is only used for syncing to, using Allway Sync, the
only program to be opened and closed in the session. Sometimes the
drive can be ejected other times it is held onto saying it is in use.


You could try Sysinternals Handle.exe or Process Explorer I think
also has the functionality of Handle in it. And see what files
are open on that volume.

Paul

On 23/04/2018 17:57:43, Paul wrote:
mick wrote:
On 23/04/2018 14:50:17, Paul wrote:
Wolf K wrote:
On 2018-04-23 07:06, Paul wrote:
Closing Explorer windows (even when it doesn't seem possible
the windows in question have a "handle" on the disk), seems
to work here.

Explorer checks all drives on boot, including external ones. The tree
view shows all powered drives. So closing the Explorer window(s) and
using the command-line should make for a faster Copy.

Or????


If you use Safely Remove, to remove a USB device, sometimes
you get a complaint the "volume is busy".

The obvious answer, is some program running on your machine,
has a file open on the volume in question, perhaps about
to write it.

Now, if you use Process Explorer or Handle.exe, you can check
to see exactly which files are open. And then close the programs
that have the files in question open.

OK, what happens if Safely Remove *still* won't remove the
drive ? It's still busy it seems.

One thing you can do, is close the File Explorer windows, as
sometimes, even though the File Explorer window isn't displaying
the volume contents in question, it still seems to have a handle
open on the drive.

Even after you've done that, there can still be cases where it
won't release so you can unplug it. One source of contention,
is the NTFS TxF feature, is keeping TxF files open on the
volume. Using the Disk Management "Offline" command seems
to help in that case.

Shutting down the machine should always release the volume,
but you might see a slow shutdown process, if the "culprit"
is gasping for breath at shutdown :-) There are some things
which don't go gracefully.

I had one of these the other day on this machine, and I
had to power cycle to regain control. I restored from backup,
because I love to do stuff like that... Just to make sure
whatever happened, didn't leave any after-effects. The
volume wasn't actually damaged, but since I had the backup,
I used it.

Paul

I do not use File Explorer and nothing is using the external disk AFAIK.
I have watched the external disk light for ages which is steady, so nothing
is writing to it.
The external disk is only used for syncing to, using Allway Sync, the only
program to be opened and closed in the session. Sometimes the drive can be
ejected other times it is held onto saying it is in use.


You could try Sysinternals Handle.exe or Process Explorer I think
also has the functionality of Handle in it. And see what files
are open on that volume.

Paul

Thanks, I'll have a look at those.

--
mick

Andy Burns wrote:
Paul wrote:

Before the destination device can be unplugged, the FIFO
queue has to be empty. If you were to use Safely Remove,
it will need to drain the FIFO queue first.

I'm getting more and more frustrated with "safely remove" it mostly
seems to claim the drive is still in use, this happens with USB mass
storage, USB attached SCSI (UASP) and eSATA drives

Various things sometime help, but none seems to help in all cases.

Closing all explorer windows
Killing all explorer processes
Closing all apps and logging out
Looking for open handles with procmon

It's as if they *want* us to just yank the cable ... grr!

I saw that in the meantime, you already have a solution (flip the
isOffline and isReadOnly bits on your backup drive), so these are just
additional suggestions:

- Hibernate the system. Quicker and less invasive than a Restart, etc..

- If your disk is a WD (Western Digital) one, have a look for a WD
program, probably in the Notifications Area of the Taskbar.
In my case it's 'WD Access' aka 'WD App Manager'. I haven't bothered
(yet) looking what it is, how I got it, how to get rid of it, etc., I
just Quit it.

HTH.

In article ,
says...

Jason wrote:

Why does the speed vary like this?

I'm assuming at those sizes, that the external USB disk isn't an SSD,
but maybe it's a hybrid SSHD? (e.g. 8GB of flash + 1TB of rust)


p.s. you've used a mixture of Mb and MB which makes talking about your
actual numbers rather awkward.

I (thought) I corrected the units...maybe missed on. It is
a garden variety WD external drive - probably 5400 rpm.

In article ,
lid says...

Isn't Windows marvelous ?

First, you have to look at your ecosystem.

What is the transfer rate of the source drive ?

What is the transfer rate of the destination drive ?
Is the destination drive or device, actually USB2, or
is a USB3 port and USB3 enclosure controller being used ?

It's a garden variety WD USB3 drive connected to a USB3
port.

The port is -not- setup for quick removal. I'll give that
a try if the culprit may be the (broken?) buffering
scheme.

btw, I am using plain ol' right-click Copy/Paste for this.
There is a more elaborate copy utility in Windows but I
can't remember its name just now. I will remember
eventually and give that a try.

Other times I've delved into Windows' file behavior I
think I learned that it does r/w in 64 kB chunks. I don't
know if that is configurable.

In article , Zaidy036
@air.isp.spam says...
Make a batch so your work time is not impacted by copy time:


I do that. Everything happens while I'm sleeping :-)

In article ,
lid says...

On 04/22/2018 08:23 PM, Jason wrote:
In article -
september.org, says...
it
proceeds at around 110 Mb/s


Mb/s should be MB/s everywhere in my original post

A lot of people mix up big B (bytes) with little b (bits).

I know and it drives ME nuts, so if I'm guilty, flog me
:-) (I thought I'd fixed it, maybe missed one...)

btw, I am using plain ol' right-click Copy/Paste for this.
There is a more elaborate copy utility in Windows but I
can't remember its name just now. I will remember
eventually and give that a try.



RoboCopy


--
Zaidy036

Jason wrote:
In article ,
lid says...

Isn't Windows marvelous ?

First, you have to look at your ecosystem.

What is the transfer rate of the source drive ?

What is the transfer rate of the destination drive ?
Is the destination drive or device, actually USB2, or
is a USB3 port and USB3 enclosure controller being used ?

It's a garden variety WD USB3 drive connected to a USB3
port.

The port is -not- setup for quick removal. I'll give that
a try if the culprit may be the (broken?) buffering
scheme.

btw, I am using plain ol' right-click Copy/Paste for this.
There is a more elaborate copy utility in Windows but I
can't remember its name just now. I will remember
eventually and give that a try.

Other times I've delved into Windows' file behavior I
think I learned that it does r/w in 64 kB chunks. I don't
know if that is configurable.

The cluster size on a partition is configurable.

The OS partition should be 4K (NTFS).
There was even a release or two of Windows 10 that would
tolerate other choices on the OS partition, but
something happened to that tolerance, and now you
should stick with 4K.

A data partition can be 64K (and when doing so for NTFS,
allows moving the drive to other Windows OSes that
understand NTFS).

On Windows 10, a larger cluster size than that is
allowed. I couldn't believe it when I saw some new
options on Win10, but my joy was short-lived when
WinXP didn't tolerate the oversized clusters.

The CRT (C language runtime) has buffered I/O, where
the maximum buffer size supported seems to be related
to the size of a cluster. I've written a couple of small
utilities where I use that, and it makes the runtime
responsible for buffering, rather than my program.
You can get at least 300MB/sec out of that.

You can always write your own buffering routine, to
do transactions in a larger size than that.

Robocopy uses the non-blocking I/O features the
system has to offer, and you can have reads and writes
overlapped during a copy.

But the egregious behaviors of Windows 10, swamp out
any nuances, so really there's no reason to try to
optimize anything. The OS decides whether you're
late for an appointment - you don't. It's like they
don't have any staff in the storage department,
watching the shop.

If I go back to a pre-Vista OS, I'm more likely to
see consistency, enough to make me tune a few things.

(Use the download button at the top of the page, to get
the original-sized image. The image as submitted is 768x998.)

https://postimg.cc/image/6wofuqb07/

To put that in perspective, the original release candidate
for Win10 had a flat-as-a-pancake transfer curve for
that test case. Too bad the transfer rate in that release,
was only *1GB/sec*. In other words, consistent... but slow.
Now, nobody really needs consistency. You're not supposed
to be looking at screens like that - "just concentrate
on updating your Facebook please, stop looking at perfmon".

Paul

Jason wrote:
In article , Zaidy036
@air.isp.spam says...
Make a batch so your work time is not impacted by copy time:


I do that. Everything happens while I'm sleeping :-)


I hope you're running that software that delays
reboots while you're away :-) (Maybe you've delayed updates
or something, and that's why it's not an issue.)

No, I'm not talking about whacked-out pages like this
(IT treadmill).

https://docs.microsoft.com/en-us/win...e/waas-restart

This thing appears to use an AutoHotKey script,
packaged as an EXE. If you run the EXE, the idea is,
if a Windows Update comes in while you're away, it
"moves the goalposts" on the Active Hours thing.

https://github.com/cryptogeek/NoReboot

Paul

Jason wrote:

I (thought) I corrected the units...

Yeah you did in your follow-up

In article ,
lid says...
it
"moves the goalposts" on the Active Hours thing.

That happened with my wife's machine. I haven't seen it
yet with this one/