Troubleshooting

Win XP Pro Laptop

I copy a set of the same files to several places; NAS and USB Flash and
USB Eternal drive.

All except copy to USB Flash seem OK but when copying to USB Flash the
last file being copied always pauses for many seconds making me wonder
what the heck I going on. Each USB Flash is 32 G and different brands.

This does not happen on a different but similar laptop.

This happens with any USB Flash drive I try to copy to.

Internet access is only through the CAT5 cable. WiFi is off.

USB connections are to
USB Drive 1
USB Drive 2
Mouse
USB Flash (one of several tried).

What should I do ?

--- news://freenews.netfront.net/ - complaints: ---

In article ,
says...

Win XP Pro Laptop

I copy a set of the same files to several places; NAS and USB Flash and
USB Eternal drive.

All except copy to USB Flash seem OK but when copying to USB Flash the
last file being copied always pauses for many seconds making me wonder
what the heck I going on. Each USB Flash is 32 G and different brands.

This does not happen on a different but similar laptop.

This happens with any USB Flash drive I try to copy to.

Internet access is only through the CAT5 cable. WiFi is off.

USB connections are to
USB Drive 1
USB Drive 2
Mouse
USB Flash (one of several tried).

What should I do ?

--- news://freenews.netfront.net/ - complaints: ---

I see that often when copying large number of larger files to external
usb flash and hard disks. I assume it's the folder info being updated as
I see it happen after pretty much every file for a brief moment that
might extend to couple of seconds. Happens just before filename changes
to next item and for last one just before dialog disappears.

I use FreeFileSync now-a-days for any large copies. Seems more reliable
than Window's copy. Just had an external start failing during a copy. I
noticed speed went WAY DOWN for little while and sure enough
FreeFileSync told me about the issue when it finished copying
everything. I suspect Windows would have just had the dialog disappear
and I'd have been no wiser to check SMART info and after seeing
reallocation errors, SeaTools (SeaGate external) failed it's long test.
Drives in mail as I write.

I often copy well into Tb range across drives and sometimes networked
systems (takes forever Had to do that for drive in mail, e.g. copy
data to another drive. 1Tb across 150,000 files took 16+ hours.
FreeFileSync did it no problem. Was glad to find "failed" drive didn't
give any read errors ... phew ... although do have a second backup on
another external and it's also all on dvd.

Buddy wrote:

Win XP Pro Laptop

when copying to USB Flash the
last file being copied always pauses for many seconds making me wonder
what the heck I going on.

You never mentioned what file system is on the USB drive. If there is
nothing you cannot reproduce on the USB drive, fast format it to NTFS
and retest. If it is already NTFS, retest after formatting as FAT32. I
don't remember which but users have reported one file system seems to
eliminate the long pause at the end of the copy; however, that just
means the copy takes longer (you'll wait during the copy versus at the
end of it).

If you use a benchmarking tool, like HD Tune (https://www.hdtune.com/),
that checks transfer speed across a variety of file sizes, you'll see
the rated speed for the USB drive is not what you'll achieve across all
file sizes. HD Tune is paywa you get to try it for free but only to
measure read speeds (and they warn the payware write tests are
destructive since, after all, you are writing to the device). See the
non-Pro (free) and Pro (payware) comparison at:

https://www.hdtune.com/download.html

Since your issue is with write speeds and a pause at the end of a copy
operation, the free version of HD Tune won't test the write speeds to
show you how your device performs. There was http://usbflashspeed.com/
but that site give "502 bad gateway", so that site is worthless. I've
never used it and didn't bother to see if it was available from
elsewhere.

The progress bar for Windows' copy has never been accurate. You're
seeing it pausing at the end because it was not that close in the first
place. When transferring files from a fast to slow source, the progress
meter is highly inaccurate. The files from the HDD/SDD are getting
cached or buffered, so the progress meter moves forward. Then the cache
gets dumped to the slow device, and the progress meter pauses. The
progress meter shows the copy is done but it could take a long time to
empty the buffer's contents to the target drive. You could disable the
write cache in Windows, but then overall drive performance would
severely suffer.

You never mentioned which type of USB ports you have. However, since it
appears you are asking about a Windows XP host (and not a Windows 7 host
despite cross-posting to that newsgroup), likely you only have USB 2.0
ports. That means it will take even longer to empty the write cache (in
the OS and in the device) making the progress meter pause longer at the
end when there is still more to copy but the meter is way off.

Some users prefer using a 3rd party copy program. For example,
Teracopy, a freemium product, has a safer (or more reliable) copy method
which means it can be slower while eliminating the long pause at the 99%
threshold exhibited by Windows own tools. Teracopy stopped being
developed and no longer supported back in October 2017, but that doesn't
mean it stopped working.

Could be an iffy spot on the drive that is getting read. Windows will
retries several times when it gets a read error, and so does the
firmware in the drive, so the two magnify each other with lots of
retries. If just one of those dozen, or more, retries succeed then the
copy succeeds, and you don't know about the iffy spots. It's when all
retries fail that the copy aborts (instead of letting you skip the
failing file and proceed with the rest). Have you run 'chkdsk /r' on
both the source drive?

Windows Explorer's copy as well as the command-line copy.exe has no
error recovery. 'xcopy' does (/c switch), as well as robocopy.exe (but
you need to shorten its /r retry count of 1 million and it /w retry wait
of 30 seconds or you might wait over a year on just 1 retry that
eventually fails). Sorry, it has been way too long since I last used
Windows XP -- plus you also *cross-posted to the Windows 7 newsgroup* --
to remember if WinXP had xcopy and robocopy. As I recall, I had to get
the Win2000 Server toolkit to get robocopy.exe from it into a WinXP
setup.

Can also be caused by anti-virus software that wants to interrogate all
those new files you just created. While the AV might have an option to
ignore removable media, that wouldn't be smart since sneakernet is a top
vector for malware infection (after e-mail and web downloads). If you
are copying huge-sized files, it can take a long time for the AV to scan
them. Some AVs will have a configurable threshold of when to cease
scanning into a file. The idea is that the malware would infect the
first part of the target file, so scanning after, say, 100MB, is more
than enough to cover the infection area. However, some malware puts
only a small bit of code at the end and the rest replaces a data block
at the end, like all those help messages coded into the program. Data
blocks are typically positioned at the end of an executable file. In
that case of malware, the first part is a stub to load the payload which
is at the end of the file. The AV might catch the stub code at the
front but then the major portion of the malware code is at the end.

What should I do ?

--- news://freenews.netfront.net/ - complaints: ---

Stop using Netfront's trial service that spamifies your posts by
appending an illegitmate signature block. Sigs start with "-- \n" (2
dashes space newline), not "--- \n") (3 dashes, space, newline). They
know that and don't want their spam hidden by users that configure their
NNTP client to hide sig blocks which are usually spam, fluff, or
off-topic.

On 2019-2-27 7:42, VanguardLH wrote:
Some users prefer using a 3rd party copy program. For example,
Teracopy, a freemium product, has a safer (or more reliable) copy method
which means it can be slower while eliminating the long pause at the 99%
threshold exhibited by Windows own tools. Teracopy stopped being
developed and no longer supported back in October 2017, but that doesn't
mean it stopped working.

I use Fastcopy (https://fastcopy.jp/en/ ), which is really fast.

--
Regards,
Lu Wei
IM:
PGP: 0xA12FEF7592CCE1EA

Buddy wrote:
Win XP Pro Laptop

I copy a set of the same files to several places; NAS and USB Flash and
USB Eternal drive.

All except copy to USB Flash seem OK but when copying to USB Flash the
last file being copied always pauses for many seconds making me wonder
what the heck I going on. Each USB Flash is 32 G and different brands.

This does not happen on a different but similar laptop.

This happens with any USB Flash drive I try to copy to.

Internet access is only through the CAT5 cable. WiFi is off.

USB connections are to
USB Drive 1
USB Drive 2
Mouse
USB Flash (one of several tried).

What should I do ?


This web page has some information on device caching.
This doesn't directly affect your analysis at the moment,
but I think it's good background reading about USB sticks
and the OS in general.

https://www.uwe-sieber.de/usbstick_e.html

*******

The first thing you need, is a reliable way of charting
low level access.

Well, there isn't one. WinXP seems to have no working
Performance Counter for USB that I could find. I looked
in the "Add Counter" section of perfmon.msc, and there's
just nothing applicable. PhysicalDisk and LogicalDisk
both ignore flash devices. I tried HDTune and 7ZIP CRC
calc as stimulus, one working at the device level,
the second working at the file level. And neither would
cause a "bump" in a perfmon.msc display.

I did find an indirect measure however. Process Monitor
from Sysinternals has fine-grained timestamps, so it
can give some idea of the I/O rate. These two pictures,
show captured traces for the two stimulus. The System
PID 4, would likely refer to some physical layer stuff.
If you take the inverse of the timestamp differences
and the length of transfer, you'll see about 120MB/sec
or so for the USB3 flash stick used in this test.

https://i.postimg.cc/d3crf6fH/procmo...tune-flash.gif

Now, when a file level operation is attempted, such as
7ZIP reading a file, I see some other behavior. Not in
this picture, mind you. This is just to show that there
*is* a way to timestamp I/O in WinXP.

https://i.postimg.cc/GmDrLjgJ/procmo...flash-file.gif

When the read operation is on-going, it seems every second
or two, 7ZIP I/O kinda slows down a bit, so there's a
"pulsing" behavior.

And in Task Manager, there's nothing in the Kernel Memory
paged and unpaged pools, that indicates WinXP is feeling
any stress. Sometimes I/O on WinXP slows to a crawl,
because some sort of "garbage collection" is done
on the pools. You may see this if transferring a
200GB file from one hard drive to another. Not everyone
sees this, so it may depend on installed software. I could
never figure out why the pools were under pressure for
what should be a "reusable buffer" situation.

On WinXP, the paged and unpaged pools are statically allocated,
and you're only allowed to use a tiny fraction of system memory
for them. Whereas on the later OSes, the pools can use
pretty much the entire memory.

To me, the first part of understanding a problem, is
collecting data. You need to run ProcMon, save the trace
as a .csv (comma separated variable text file suited
for LibreOffice Calc or MS Excel). There, you might process
the trace data, to make yourself a fancy graph. Is that
easy to do ? Well, not really :-/ But it can be done,
if you need answers.

*******

AFAIK, WinXP doesn't have a System Write cache. It has
a System Read cache, and when benchmarking, you have to
make sure the "objects" being transferred, are larger
than the amount of slack memory present in the system,
to help "invalidate" the cache. When benchmarking, you
want to make sure no cache is "cheating" the benchmark
of getting real results. I might read (checksum) an
8GB single file, to help flush the cache before
doing a copy test to the USB stick.

WinXP has a System Read cache and no System Write cache.
Vista+ has both, and it's mighty annoying, since you
dare not pull out any storage devices if gigabytes
of data are still "draining" from the Write cache.
You could lose up to 50 seconds worth of data, by pulling
a USB flash prematurely. File Explorer says the transfer
is stopped in Vista+, while in fact the cache is still
draining into the device. And on Vista+, you might have
a perfmon.msc performance counter, but maybe the
Process Monitor ETW trace would have trouble watching
the cache draining part. I don't know what that
behavior is "charged to", whether it's a PID 4 system
process or what.

Summary: WinXP poorly instrumented
WinXP has no "big" system write cache (good!)
WinXP has undersized paged and unpaged pools (bad!)
I can't really guess what is causing your problem,
but the OS design is simple enough, it should not
be the source of the problem. There's more complexity
on Vista+, to make analysis more difficult

I can see some "delays" when I do a 7ZIP checksum operation,
which means 7ZIP is "sharing" the machine with all my
other running tasks. When you do your testing, try to
make sure no "hogs" like a web browser are open, and
that the CPU isn't railed. I'm getting at least 20% CPU
on some of my test cases.

If you have an AV program, those are an unchecked menace,
and can mess up just about anything you're trying to do.
Do the two test machines use different AV programs ?

Paul