A Windows XP help forum. PCbanter

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » PCbanter forum » Windows 10 » Windows 10 Help Forum
Site Map Home Register Authors List Search Today's Posts Mark Forums Read Web Partners

performance when copying large files



 
 
Thread Tools Rate Thread Display Modes
  #1  
Old May 21st 18, 03:58 AM posted to alt.comp.os.windows-10
Jason
external usenet poster
 
Posts: 144
Default performance when copying large files

This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?

Ads
  #2  
Old May 21st 18, 04:27 AM posted to alt.comp.os.windows-10
Paul[_32_]
external usenet poster
 
Posts: 11,873
Default performance when copying large files

Jason wrote:
This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?


I've seen this before, and one of the symptoms seems to be
the system is leaking paged pool. And a garbage collector tries
to scrape enough paged pool together to keep the system running.

Modern Windows OSes allow a lot more of system memory to be
used as Paged Pool, but this still won't prevent such behaviors
from happening (eventually).

The strange part, is what is leaking pool, and what combination
of drivers does this ? I've never been able to definitely trace
this down. I think I tried running PoolMon once, and didn't manage
to reproduce the problem.

The problem can become so bad, that the system will suffer
"Delayed Write Failure" in the Event Log (that's if the Event Log
can even be updated). You could probably simulate this situation,
by using the evaluation version of the NeatVideo plugin, which
used to attempt to limit customer usage of the evaluation
version, by leaking pool on purpose. And you'll see the system
become slower and slower. If you're not careful, you won't return
to the system at the right moment in time, and your
opportunity to run Task Manager and kill it, will be
lost. Then, you have to use the power button to recover.

The symptoms I've seen a

1) Report of increased page pool occupancy.
(Easier to see in the WinXP Task Manager)
2) Declining write rate. Until eventually writes
begin to time out, and an Event Viewer log entry
is created. It might take 12 hours for it to get
this bad.
3) Increasing CPU activity, as the OS attempts to garbage
collect released pool memory.

Even if I knew which application was doing it, that would
be a start. But the thing is, drivers left behind in
a system, can continue to be a cause of the issue, without
the program in Ring3 running at the time. Drivers can
cause problems "just by being there" when they're filter
drivers and all the data flows through them anyway.

Acronis and Macrium install services. Acronis is the poster
boy "for adding stuff", because they love to dress up
their products with an excess of services added (they will
add more than one). I never get the feeling everything
Acronis adds, is actually there for a reason.

Another potential source of conflict, could be virtual
machine hosting software. It's a wild possibility.

Paul
  #6  
Old May 21st 18, 07:58 AM posted to alt.comp.os.windows-10
mike[_10_]
external usenet poster
 
Posts: 1,073
Default performance when copying large files

On 5/20/2018 7:58 PM, Jason wrote:
This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?

I've seen odd slowdowns using file manager.
Same copy using totalcommander doesn't have that problem.

How fast is your external drive?
I'm assuming you meant 2TB??
When I copy to external media, it starts fast, but then drops
to a much lower speed somewhere around the 1GB transferred point.
Pausing for a while restores the speed.
I'm theorizing that the fast speed is how fast the internal drive
can write to a buffer. The slow speed is the speed that the external
drive can accept the data.
This is exceptionally obnoxious with 256GB USB thumb drives.
  #7  
Old May 21st 18, 08:40 AM posted to alt.comp.os.windows-10
Paul[_32_]
external usenet poster
 
Posts: 11,873
Default performance when copying large files

Jason wrote:
This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?


You know, rather than cook up some corny
explanation, we'd better be careful here that
this isn't "ordinary" sharing of a resource.

An Intel chipset (PCH), has a set-of-six ports off
one 60MB/sec USB2 controller and a set-of-eight ports
off a second 60MB/sec controller. The bandwidth is
shared inside a group. That's an example of some
sharing going on.

Your Imgur picture looks to be going faster than
that, so that's not it.

I don't know how, or whether, USB3 does sharing. While
we're all used to seeing ~400MB/sec transfers from
a single USB3 device, what would happen if your
config had some sort of sharing going on ?

Have a look at what ports are being used
in this scenario of yours, so we don't
end up postulating space aliens for nothing :-)

Even some HDTune pictures of the source device
and the destination device, might provide you with
some hints of how the stuff works.

Paul
  #8  
Old May 21st 18, 09:37 AM posted to alt.comp.os.windows-10
Andy Burns[_6_]
external usenet poster
 
Posts: 1,318
Default performance when copying large files

Paul wrote:

Modern Windows OSes allow a lot more of system memory to be
used as Paged Pool, but this still won't prevent such behaviors
from happening (eventually).

The strange part, is what is leaking pool, and what combination
of drivers does this ? I've never been able to definitely trace
this down. I think I tried running PoolMon once, and didn't manage
to reproduce the problem.


I did manage to identify leaky drivers a few times with poolmon, but
that was back in the days of NT4 and W2K, not needed it for many years.

  #10  
Old May 22nd 18, 12:32 AM posted to alt.comp.os.windows-10
Jason
external usenet poster
 
Posts: 144
Default performance when copying large files

In article , ham789
@netzero.net says...

On 5/20/2018 7:58 PM, Jason wrote:
This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?

I've seen odd slowdowns using file manager.
Same copy using totalcommander doesn't have that problem.

How fast is your external drive?
I'm assuming you meant 2TB??
When I copy to external media, it starts fast, but then drops
to a much lower speed somewhere around the 1GB transferred point.
Pausing for a while restores the speed.
I'm theorizing that the fast speed is how fast the internal drive
can write to a buffer. The slow speed is the speed that the external
drive can accept the data.
This is exceptionally obnoxious with 256GB USB thumb drives.


Opps. Yes, TB not GB.
It's a USB3 drive. When things are clipping along the copy
proceeds at 100MB/s +/- a little. I see the slowdown start
very early into the transfer. The graph shown is of a 450+
GB transfer. I haven't tried pausing it and re-starting.
I'll see if that has an effect.
  #11  
Old May 22nd 18, 01:30 AM posted to alt.comp.os.windows-10
pjp[_10_]
external usenet poster
 
Posts: 1,183
Default performance when copying large files

In article ,
says...

In article , ham789
@netzero.net says...

On 5/20/2018 7:58 PM, Jason wrote:
This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?

I've seen odd slowdowns using file manager.
Same copy using totalcommander doesn't have that problem.

How fast is your external drive?
I'm assuming you meant 2TB??
When I copy to external media, it starts fast, but then drops
to a much lower speed somewhere around the 1GB transferred point.
Pausing for a while restores the speed.
I'm theorizing that the fast speed is how fast the internal drive
can write to a buffer. The slow speed is the speed that the external
drive can accept the data.
This is exceptionally obnoxious with 256GB USB thumb drives.


Opps. Yes, TB not GB.
It's a USB3 drive. When things are clipping along the copy
proceeds at 100MB/s +/- a little. I see the slowdown start
very early into the transfer. The graph shown is of a 450+
GB transfer. I haven't tried pausing it and re-starting.
I'll see if that has an effect.


Don't those things use some type of buffer? I'd assume once buffer is
full and keeps being filled the actual writing would slow down. Small
file and buffer doesn't get filled, large file and it's continually
being filled.
  #12  
Old May 22nd 18, 02:21 AM posted to alt.comp.os.windows-10
Paul[_32_]
external usenet poster
 
Posts: 11,873
Default performance when copying large files

pjp wrote:
In article ,
says...
In article , ham789
@netzero.net says...
On 5/20/2018 7:58 PM, Jason wrote:
This has been discussed quite a lot before, but it
continues to interest me. I create a full image backup of
my system once a week and an incremental backup each of
the next six days. Backups (Acronis) go on a spare HD
partition and I then copy them to an external (WD)
Passport 2GB drive. Copying the full backup image takes
over an hour. What I've noticed, usually, but not every
time (!?) is that the transfer gradually slows down. See:
https://imgur.com/hESFJq1

While the copy proceeds there is no other activity beyond
the usual Windows background stuff running - no apps,
browsers, etc.

What might account for this?

I've seen odd slowdowns using file manager.
Same copy using totalcommander doesn't have that problem.

How fast is your external drive?
I'm assuming you meant 2TB??
When I copy to external media, it starts fast, but then drops
to a much lower speed somewhere around the 1GB transferred point.
Pausing for a while restores the speed.
I'm theorizing that the fast speed is how fast the internal drive
can write to a buffer. The slow speed is the speed that the external
drive can accept the data.
This is exceptionally obnoxious with 256GB USB thumb drives.

Opps. Yes, TB not GB.
It's a USB3 drive. When things are clipping along the copy
proceeds at 100MB/s +/- a little. I see the slowdown start
very early into the transfer. The graph shown is of a 450+
GB transfer. I haven't tried pausing it and re-starting.
I'll see if that has an effect.


Don't those things use some type of buffer? I'd assume once buffer is
full and keeps being filled the actual writing would slow down. Small
file and buffer doesn't get filled, large file and it's continually
being filled.


Modern hard drives have a 64MB or 128MB cache on the drive
controller board. For a sustained transfer scenario, that buffer
could fill in under a second (the buffer could fill if the
input rate exceeded the platter rate, subject to the tagged
buffer limits of AHCI).

The System Write Cache can be larger. It seems to use a maximum
of a certain percentage of the system RAM. Like maybe 8% or so.
You can watch the available system RAM drop, if the source storage
device is faster than the destination storage device.

And the System Write Cache isn't that smooth either. It doesn't drain
in as predictable a way as it should. But it still doesn't account
for the transfer curve in the picture.

You can use perfmon.msc and select some disk read and disk
write counters, to get a "second opinion" on what the system
is actually doing at the time. There is also an option in the
snap-in, to log the counters to a text file, for later study
(curve fitting or whatever).

A hard drive is 2x as fast near the outer circumference,
than near the hub, but I hope that's not what we're seeing
in that transfer curve. You can use HDTune to convince
yourself what the curve looks like when the entire drive
is being used up. I have drives here, that start at 200MB/sec
on the outer diameter, and finish up at 100MB/sec near the
hub. And the perfmon.msc graph doesn't show enough minutes of the
transfer, to show that characteristic in one view. Whereas
the File Explorer transfer curve shows info for the
entire period of the transfer (even if it exaggerates
the beginning of the transfer, when it's filling the
write cache or something).

But these are all effects you've seen before, so none
of them should be a real surprise. Windows is like
"file transfer theater" when it comes to this stuff :-)

The OPs picture looks a bit too flat and "straight line"
to be a disk circumference issue. If the decline in speed
showed some overall curve to it, just the hard drive
characteristic might explain it. I would need to know:

1) Disk fill before transfer (assumes files are packed to
the left on the partition, but they could really be
anywhere in actual conditions). Like say "I had
100GB on the 500GB USB3 hard drive before I started".

2) Size of transfer - "I was adding a 250GB file to the 100GB
of files already on the drive".

That would make it easier to justify the decline in transfer
rate being linear, if the sampling we're seeing in the
File Transfer dialog is showing us just a small portion of
the entire disk surface transfer speed curve.

If you had a 500GB short-stroked drive, those can almost
maintain their transfer rate over the entire surface.
WDC shipped a few of those, when closing out the 500GB 3.5"
business. They were then using 1TB drives, and only
activating the first half of the circumference (the
faster "outer donut"), rather than continuing to make
500GB drives. And when you do that on a drive,
you get to keep the best part of the transfer curve.
But WDC wasn't consistent, because two drives in the
pile at the computer store, one was short stroked
and the second one wasn't. And the serial numbers
didn't betray anything. I had no prior warning that's
what I was getting - this is not in the spec sheet.

The one on the right here, shows the regular 2:1
ratio between the outer circumference and the inner
circumference. The one on the left is a short-stroked
drive 1TB in size, where only the outer 500GB is used.
The the speed drops from about 175MB/sec to 155MB/sec
over the available surface.

http://s29.postimg.cc/8b7cj872v/wd500gb.gif

Paul
 




Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off






All times are GMT +1. The time now is 04:54 PM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 PCbanter.
The comments are property of their posters.