External Drive On USB3 Seems TB Dragging Down Mb/s On SATA Drive ?

Yes, there was something wrong with the drive.....

CHKDSK /F fixed that ....

But, even with the drive performing normally, a massive delete against
it (it holds Alert snapshots from a bunch of IP cams) brings Macrium
Reflect (which is reading from a NAS share and writing to another drive)
to it's knees - as in 300 Mbp/s before and 1 Mb/s during/after.

But the problem drive lives in a USB3 enclosure plugged in to a USB3
port while "innocent bystander" drive which is Reflect's target, is
internal and directly connected to one of the mobo's SATA ports.

And the really strange thing is that the degradation of Reflect's write
activity against the "innocent bystander" drive remains - even after the
Delete operation on the problem drive had finished, and even after the
problem drive has been powered off. Reflect just keeps crawling along
at less than 1 Mb/s.

LAN SpeedTest against the "innocent..." drive returns the expected
gigabit+ speeds and returns about 400 Mb/s reading from the NAS share
that Macrium is reading.

Finally, about 20 minutes after powering the problem drive has been
powered down, Reflect's Mb/s suddenly climbs back up from 1 Mb/s to the
usual 275-300 Mb/s.

OK.... seems like Reflect cannot be totally innocent here because of the
high speeds reported by LAN SpeedTest against both Reflect's source and
it's target while Reflect was crawling along at 1 Mb/s.

Reflect's problems aside, can anybody shed some light on why something
with a USB-connected drive should affect operations against a
SATA-connected drive?

Seems like one of those "Learning Opportunities".
--
Pete Cresswell

(PeteCresswell) wrote:
Yes, there was something wrong with the drive.....

CHKDSK /F fixed that ....

But, even with the drive performing normally, a massive delete against
it (it holds Alert snapshots from a bunch of IP cams) brings Macrium
Reflect (which is reading from a NAS share and writing to another drive)
to it's knees - as in 300 Mbp/s before and 1 Mb/s during/after.

But the problem drive lives in a USB3 enclosure plugged in to a USB3
port while "innocent bystander" drive which is Reflect's target, is
internal and directly connected to one of the mobo's SATA ports.

And the really strange thing is that the degradation of Reflect's write
activity against the "innocent bystander" drive remains - even after the
Delete operation on the problem drive had finished, and even after the
problem drive has been powered off. Reflect just keeps crawling along
at less than 1 Mb/s.

LAN SpeedTest against the "innocent..." drive returns the expected
gigabit+ speeds and returns about 400 Mb/s reading from the NAS share
that Macrium is reading.

Finally, about 20 minutes after powering the problem drive has been
powered down, Reflect's Mb/s suddenly climbs back up from 1 Mb/s to the
usual 275-300 Mb/s.

OK.... seems like Reflect cannot be totally innocent here because of the
high speeds reported by LAN SpeedTest against both Reflect's source and
it's target while Reflect was crawling along at 1 Mb/s.

Reflect's problems aside, can anybody shed some light on why something
with a USB-connected drive should affect operations against a
SATA-connected drive?

Seems like one of those "Learning Opportunities".

You have interrupts and remaining percentage CPU
as possible bottlenecks.

I would not typically expect to see what you've seen.
For the most part, the I/O ports are independent. (Sharing
IDE drives on a ribbon cable isn't independent, but SATA
is.) USB controllers can be shared, so doing activities
on two USB ports could cause a throttling effect.

But I would have to guess interrupts, or a less than
stellar error handling when the OS deals with CRC
errors on the drive in the USB enclosure. On a bad block,
a hard drive can "spin its wheels" for 15 seconds, before
returning the bad status of a block, which may cause
the kernel to block depending on how the code is written.

And the Macrium bandwidth measurement could be a long term
average, rather than an instantaneous one. You can use
Performance Monitor if you need to review what is going
on. Even the read bytes and write bytes in Task Manager
is better than nothing, as a metric. You're not limited
to the Macrium transfer indicator - there are others.

Paul

En el artículo ,
(PeteCresswell) escribió:

Reflect's problems aside, can anybody shed some light on why something
with a USB-connected drive should affect operations against a
SATA-connected drive?

USB is very processor intensive. You're deleting a pile of files on a
USB-connected drive which takes up a lot of CPU and has a knock-on
effect on operation of the system as a whole, not just your SATA-
connected drive.

--
(\_/)
(='.'=) Bunny says: Windows 10? Nein danke!
(")_(")

PeteCresswell wrote:

a massive delete against [the drive] (it holds Alert snapshots from a
bunch of IP cams) brings Macrium Reflect (which is reading from a NAS
share and writing to another drive) to it's knees - as in 300 Mbp/s
before and 1 Mb/s during/after.

But the problem drive lives in a USB3 enclosure plugged in to a USB3
port while "innocent bystander" drive which is Reflect's target, is
internal and directly connected to one of the mobo's SATA ports.

And the really strange thing is that the degradation of Reflect's write
activity against the "innocent bystander" drive remains - even after the
Delete operation on the problem drive had finished, and even after the
problem drive has been powered off. Reflect just keeps crawling along
at less than 1 Mb/s.

LAN SpeedTest against the "innocent..." drive returns the expected
gigabit+ speeds and returns about 400 Mb/s reading from the NAS share
that Macrium is reading.

Finally, about 20 minutes after powering the problem drive has been
powered down, Reflect's Mb/s suddenly climbs back up from 1 Mb/s to the
usual 275-300 Mb/s.

OK.... seems like Reflect cannot be totally innocent here because of the
high speeds reported by LAN SpeedTest against both Reflect's source and
it's target while Reflect was crawling along at 1 Mb/s.

Reflect's problems aside, can anybody shed some light on why something
with a USB-connected drive should affect operations against a
SATA-connected drive?

Seems like one of those "Learning Opportunities".

So you have 2 drives inside the same enclosure. One is connected via
USB and the other is connected via SATA. Is that correct? If so, does
the SATA cable go directly to the SATA drive inside the shared enclosure
or does the SATA cable connect to a converter board that also does the
USB-to-SATA conversion for the other drive?

When you do the massive deletes, did you disable the Recycle Bin? It
works by first-in-first-out order. If there is not enough room in the
Recycle Bin to delete the next file, it then removes the first deleted
file. If there still is not enough room in the Recycle Bin for the file
to delete, the next file gets purged from the Recycle Bin. Eventually
enough files get purged from the Recycle Bin to allow room for the file
to delete. And the process continues that way while slowing file system
access.

The Recycle Bin will severely slow your massive deletes. If you really
don't need to restore those deleted files, disable the Recycle Bin on
that drive. You'll find a 15,000 file delete that took 20 minutes with
the Recycle Bin enabled will finish in a couple seconds with the Recycle
Bin disabled (those are just example comparisons). However, disabling
the Recycle Bin only affects the one drive where is the $Recycle.Bin
folder for that drive so I can't see how 2 separate drives would be
affected by a massive delete on only one of them.

How are you performing the massive delete? Some programs are
synchronous in that they do not exit until the operation has completed.
Some are asynchronous in that they will run the delete operation despite
they appear to have exited (just their GUI disappeared). For example, I
was going to use FreeFileSync in a batch file. After a file sync to a
temp folder, I was going to zip those files and move the .zip file to a
folder tracked by OneDrive so that a copy of it would be online in my
OneDrive account. Alas, when I told FreeFileSync to run a previously
defined template (to define what to sync and how and a bunch of other
options), the GUI immediately exited and started a separate process that
appeared as a tray icon. The result was an asynchronous file sync
operation: the program exited (but was still doing the file sync in the
background) so the next commands in the batch file would execute.
Without waiting until the sync completed, the rest of the commands were
doing moves, deletes, and other stuff before they should.

So what do you use for the massive delete? Some wipe function in
Reflect? I don't remember if it is FreeFileSync or SyncBackFree but one
of them has an option to wipe files. It does it by queuing up the files
and doing individual deletes, one delete per file, like in a loop. It
is a quick secure wipe so it focuses on one file at a time. Since the
delete operation does not use the Recycle Bin at all, whether the
Recycle Bin was enabled or disabled would not affect the secure delete
speed performed on each file in turn. For about 8000 files, the "quick
wipe" (a simple secure delete) took something like 10 minutes. I sat
there watching a list of files roll by as each got securely deleted.
Luckily most of the files were small so there weren't many bytes to zero
out.

If you are using some function within Macrium Reflect to delete the
files, you might want to instead use the pre- and post-commands to do
the deletes. Some backup programs (usually payware) allow you to
specify a command to run before and run after a backup job. I've used
them, for example, to enable an HDD, change a policy to allow access to
the HDD, and assign a drive letter to it so the backup job can find and
use the backup partition on that HDD. After the backup job complete, I
have the backup program run a post-command (batch file) that unassigns
the drive letter, sets the policy to block access to the device, and
disable the HDD. That makes for a much smaller window of opportunity
for malware to get at my backups. Acronis TrueImage and Paragon Backup
& Restore do something similar with their secure "zones" (because
programmers that left Acronis went to Paragon so the same feature showed
up at Paragon). They don't assign a drive letter and use a special
partition type number that is not recognized to help keep malware or
ignorant users with typical disk tools from deleting or encrypting the
backups. Alas, I found both to be unreliable during restores, like
bitching they cannot find the backups in hidden partition until I retry
many times until the backup program finally succeeds in accessing the
hidden partition. Many backup programs don't even have a means of
protecting the backups as they are in the file system on a drive with a
letter or networked drive that many malwares can reach. So I use the
pre- and post-commands to help hide my backups but there is a window of
opportunity during the backup job.

So you might want to use a different program, even a batch file with a
del[ete] command, instead of whatever Reflect is offering for file
deletes. Then specify that program or batch file as a pre-command to
the backup job.

First I'd disable the Recycle Bin on any drive where you don't need to
restore deleted files. Obviously the file(s) have not been deleted,
just moved elsewhere in the file system with visibility reduced. Only
if you push more files into the Recycle Bin than it can hold or by
disabling the Recycle Bin are you actually doing file deletes. If that
doesn't help, find a delete tool that does not exit before it has
actually completely finished performing all deletes. The folks in the
alt.comp.freeware newsgroup might know a freeware tool that lets you
specify what to delete but bypasses the Recycle Bin if enabled. There
have been several times when I wanted something that would do a delete,
bypasses the Recycle Bin, recursively takes ownership (by me) of all
files and folders, and sets permissions so I have full control, and
removes any open handles on the files and folders - so when I choose to
delete then it really happens.

I've heard (but not used) that TeraCopy is also fast on file deletes. I
don't know if it has a CLI (command-line interface) to let you run it as
a pre-command to a backup job or in a batch file where you could run it
before you use a command to start the backup job. I also do not know if
TeraCopy operates synchronously or asynchronously (so following commands
get executed only after TeraCopy's operation has fully completed). I
only saw TeraCopy mentioned as a fast[er] method of doing massive
deletes. It's free for personal-use only.

Nirsoft has their nircmd tool that has an "emptybin" directive to empty
the Recycle Bin. That would let the following deletes happen much
quicker because there would be no inchworming through the contents of
the Recycle Bin to make room for newer deletes. Of course, if you
already had a ton of files in the Recycle Bin then emptying it takes
awhile if the operation purges each file one at a time from the Recycle
Bin. Deleting the Recycle bin ("attrib -h -s drive:$Recycle.Bin"
followed by "rmdir /q /s drive:$Recyce.Bin") would be a lot faster.
It empties the Recycle Bin on all drives, not on just the one where you
will doing your massive deletes. Of course, the speedup from starting
with a clean Recycle Bin will last only until the Recycle Bin gets
filled up again.

You can run CCleaner from the command line (path\CCleaner.exe" /AUTO)
and it will do all the cleanup it is configured to do. It will clean
out the Recycle Bin but will take time to do other cleanup, like empty
the TIF for the web browser(s). Doing any Recycle Bin work before the
backup job will take time so your backup job will take longer. If the
backup takes 2 hours, a couple minutes, or less, to empty the Recycle
Bin won't make much difference.

Per VanguardLH:

So you have 2 drives inside the same enclosure. One is connected via
USB and the other is connected via SATA. Is that correct?

No: The "innocent bystander" drive is one of my backup drives and is in
a SATA sled connected directly to one of the mobo's SATA receptacles.

The problem drive is in an enclosure and connected via USB3.... but
again, directly to a mobo receptacle. i.e. there are no USB or SATA
cards involved... only mobo receptacles.

When you do the massive deletes, did you disable the Recycle Bin?

No... but that's on my short list now that you have said it.

And, while I was at it, also disabled "Allow files on this drive to have
contents indexed in addition to file properties"


How are you performing the massive delete?

..BAT file:
---------------------------------------------
DEL B:\Alerts /F /Q
DEL B:\Clips /F /Q
DEL B:\Clips_Stored /F /Q
DEL B:\DB /F /Q

ECHO OFF
ECHO .
ECHO .
ECHO . ------------ Done! -------------
ECHO .
ECHO .
PAUSE
---------------------------------------------

--
Pete Cresswell

Per (PeteCresswell):
No... but that's on my short list now that you have said it.

After right-clicking the recycle bin, selecting "Properties", and
turning off recycling for the drive in question (this for the benefit
of anybody else that's going there....).... I figured I had better Empty
the Recycle Bin instead of just hoping that turning it off would
suffice.

OOPS !..... "Deleting 19,093 items (48.5 GB).....11 items per second...
Items remaining: 16,082..... Time remaining: About 1 hour !"...

NB: Those numbers do not compute because (16,082/11)/60 = 24 minutes,
not 60.... but it's time estimate seems to jump around
minute-to-minute... now it's saying 30 minutes with 10,000 items to go
at 15 items/second....

Anyhow.....I am thinking that the Recycle Bin could have been a root
cause. Since down-sizing that drive from 3 TB to 1 TB (needed a 3 TB
drive for something else) it has routinely become filled to capacity by
the time I go to purge it... and, with the cam server beating on it
continuously, that's got to be creating some sort of bottleneck....
--
Pete Cresswell

On Thu, 05 Nov 2015 17:55:25 -0500 "(PeteCresswell)" wrote
in article
OOPS !..... "Deleting 19,093 items (48.5 GB).....11 items per second..


I don't understand why, but having a ton of stuff in the Recycle Bin
seems to always have a deleterious effect on performance. I have a few
friends who periodically complain to me about how their Windows machines
have slowed down and the first question I always ask is if they have
emptied Recycle. Most times, the answer is No and then Wow! after they
do...

PeteCresswell wrote:

Per VanguardLH:

How are you performing the massive delete?

DEL B:\Alerts /F /Q
DEL B:\Clips /F /Q
DEL B:\Clips_Stored /F /Q
DEL B:\DB /F /Q

Those would use the Recycle Bin as I mentioned: FIFO. Even if you
emptied the Recycle Bin, you'll eventually fill it up and run into the
FIFO slowdown again. If you want to keep the Recycle Bin enabled, there
are utilities that will do direct disk deletes (i.e., the deletes don't
go through the Recycle Bin).

Also, the DEL command does not step through a for-loop to process each
file. It builds a long list of files that it will delete. I have seen
where the list gets so long that DEL errors with some "too long" message
(and I'm not talking about directories being too deep).

You'll find deletes go extremely quickly when not using the Recycle Bin.
After spending 22 minutes to copy a folder (and its subfolders)
containing over 4700 files that occupy 23 GB, a normal DEL going through
the Recycle Bin took several minutes (I forgot how long) while a DEL
with the Recycle Bin disabled took under 3 seconds.

PeteCresswell wrote:

After right-clicking the recycle bin, selecting "Properties", and
turning off recycling for the drive in question (this for the benefit
of anybody else that's going there....).... I figured I had better Empty
the Recycle Bin instead of just hoping that turning it off would
suffice.

OOPS !..... "Deleting 19,093 items (48.5 GB).....11 items per second...
Items remaining: 16,082..... Time remaining: About 1 hour !"...

NB: Those numbers do not compute because (16,082/11)/60 = 24 minutes,
not 60.... but it's time estimate seems to jump around
minute-to-minute... now it's saying 30 minutes with 10,000 items to go
at 15 items/second....

Anyhow.....I am thinking that the Recycle Bin could have been a root
cause. Since down-sizing that drive from 3 TB to 1 TB (needed a 3 TB
drive for something else) it has routinely become filled to capacity by
the time I go to purge it... and, with the cam server beating on it
continuously, that's got to be creating some sort of bottleneck....

Kill the slow delete loop. Delete the drive:$Recycle.Bin folder
(you'll probably have to remove the hidden and system file attributes).
Then let Windows recreate the folder (if you left the Recycle Bin
enabled).

The estimated time to completion calculated by ANY program has never
been accurate. It's just a guess. Would you want it to tell you that
it will take an hour or to have no indication and be waiting 8 hours? A
better calculation is just to see how many files remain to get purged.
As you've seen, that is a slow process. The FIFO procedure to make room
in the Recycle Bin is slow. Removing items within it is slow.

I use CCleaner to clean up my disks on a periodic basis. I have a
shortcut in a toolbar shown in the Windows taskbar that I click to
perform an immediate cleanup. Most times after a web browser session, I
run CCleaner to make sure everything that I configured the web browser
to purge on its exit actually does get deleted. The shortcut runs:

"C:\Program Files\CCleaner\CCleaner.exe" /AUTO

I even have this command scheduled to run early in the morning (before
my backups run) but configure the event to wait for the computer to go
idle for awhile, like 30 minutes. That ensures that I'm not doing
something, like installing a program that puts files in the temp folder.
While not a sound practice, often installers put files under the temp
folder, especially if they need to reboot to complete the install. If
CCleaner wipes the temp folder during an install, obviously the install
is going to hang, crash, or do something nasty with its legs chopped off
and bleeding profusely.

Per VanguardLH:

You'll find deletes go extremely quickly when not using the Recycle Bin.
After spending 22 minutes to copy a folder (and its subfolders)
containing over 4700 files that occupy 23 GB, a normal DEL going through
the Recycle Bin took several minutes (I forgot how long) while a DEL
with the Recycle Bin disabled took under 3 seconds.

Just confirmed that a few minutes ago.

Just a second or two max....

Thanks for that one !

I don't guess there is an alternative to the DEL command.... ?

Or... how about DEL against the parent folder and then re-create the
folder... Would that take any less work ?
--
Pete Cresswell

PeteCresswell wrote:

Per VanguardLH:

You'll find deletes go extremely quickly when not using the Recycle
Bin. After spending 22 minutes to copy a folder (and its subfolders)
containing over 4700 files that occupy 23 GB, a normal DEL going
through the Recycle Bin took several minutes (I forgot how long)
while a DEL with the Recycle Bin disabled took under 3 seconds.

Just confirmed that a few minutes ago. Just a second or two max....
Thanks for that one !

I don't guess there is an alternative to the DEL command.... ?
Or... how about DEL against the parent folder and then re-create the
folder... Would that take any less work ?

Only if you first disable the Recycle Bin. Or you use a utility that
does direct disk (well, direct file system) deletes. I remember using
FreeFileSynce to drag in an folder which would list all the files and
subfolders under it and then using a "quick delete" which, as I recall,
was a basic secure wipe. I would think any secure wipe tool would
bypass the Recycle Bin, too.

I suspect there is a registry setting as to whether the Recycle Bin is
enabled or not but I haven't investigated that. You could use a
registry monitor to change the Recycle Bin (toggle from enabled to
disabled) to see what changed in the registry. Then you could use a
..reg file (e.g., regedit.exe /s rc-disable.reg) to disable the Recycle
Bin before the 'del' command and then reenable the Recycle bin afterward
(e.g., regedit.exe /s rc-enable.reg).

I found http://www.pctools.com/guides/registry/detail/943/ but haven't
delved into it.

On 11/6/2015 6:24 PM, (PeteCresswell) wrote:
Per VanguardLH:

You'll find deletes go extremely quickly when not using the Recycle Bin.
After spending 22 minutes to copy a folder (and its subfolders)
containing over 4700 files that occupy 23 GB, a normal DEL going through
the Recycle Bin took several minutes (I forgot how long) while a DEL
with the Recycle Bin disabled took under 3 seconds.

Just confirmed that a few minutes ago.

Just a second or two max....

Thanks for that one !

I don't guess there is an alternative to the DEL command.... ?

Or... how about DEL against the parent folder and then re-create the
folder... Would that take any less work ?


Nirsoft may help with: nircmd.exe" EmptyBin

Per Paul:
You have interrupts and remaining percentage CPU
as possible bottlenecks.

Can you aim me at something that will get me started on the Interrupt
possibility ?

I am having more problems - basically anything concerned with opening
files comes to a halt - yet Process Lasso says that virtually no CPU is
being used (http://tinyurl.com/oqt887j - window dimmed bco "Not
Responding"... I had to take a pic with my cell phone).
--
Pete Cresswell

Zaidy036 wrote:

On 11/6/2015 6:24 PM, (PeteCresswell) wrote:
Per VanguardLH:

You'll find deletes go extremely quickly when not using the Recycle Bin.
After spending 22 minutes to copy a folder (and its subfolders)
containing over 4700 files that occupy 23 GB, a normal DEL going through
the Recycle Bin took several minutes (I forgot how long) while a DEL
with the Recycle Bin disabled took under 3 seconds.

Just confirmed that a few minutes ago.

Just a second or two max....

Thanks for that one !

I don't guess there is an alternative to the DEL command.... ?

Or... how about DEL against the parent folder and then re-create the
folder... Would that take any less work ?


Nirsoft may help with: nircmd.exe" EmptyBin

Already mentioned in a prior reply of mine (but might get missed in my
verbose post), as well as using CCleaner from the command line.
TeraCopy can delete without going through the Recycle Bin but I don't
recall if it has a CLI (command line interface). Unlike TeraCopy or
CCleaner, nircmd does not have to be installed, just copied into a
folder (and optionally added to the user or system PATH environment
variable to call from anywhere without needing to specify the path to
it). I even mentioned possibly using a .reg file to disable the Recycle
Bin before the deletes and another .reg to reenable the Recycle Bin
afterward. Pete might just leave the Recycle Bin permanently disabled
for drives where he doesn't care to undelete.

(PeteCresswell) wrote:
Per Paul:
You have interrupts and remaining percentage CPU
as possible bottlenecks.

Can you aim me at something that will get me started on the Interrupt
possibility ?

I am having more problems - basically anything concerned with opening
files comes to a halt - yet Process Lasso says that virtually no CPU is
being used (http://tinyurl.com/oqt887j - window dimmed bco "Not
Responding"... I had to take a pic with my cell phone).

The Sysinternals.com Process Explorer has some info at
the top of the main window.

And if you're not using any CPU, it's unlikely to be
Interrupts or DPC. DPC are interrupt service routines
that run at non-interrupt level.

Do you have two AV programs fighting over scanning something ?

Does the disk drive have clean "SMART" stats, or
does a SMART display show some red colored entries
indicating trouble ? HDtune 2.55 has a SMART window
you can use for hard drives. That old version doesn't
have the right display info for SSDs necessarily. There
are probably other SMART display programs you can use
for a check.

You could also check Event Viewer, and see if there
are any "Delayed Write failure" or similar, which is
a sign of some sort of serious storage problem. I've had
that happen, when the OS ran out of paged pool and just
about everything on the computer was dog-slow.

Paul