Defragging System Volume Information

mick wrote:

I have had numerous computers running XP, Vista and Win 7. On the XP
and Vista systems I used Diskeeper for defragging continually in the
background. I cannot say whether Diskeepers' claims of using their
software gives faster performance or not as I did not experience any
slowness of any computers.

What I have done on the Win 7 computer is to set it up differently. I
have partitioned the hard drive with 100GB for the operating system and
programs, and 500GB for personal files, at the moment there is 48 and
350GB of files on these partitions. I have also opted to use free
software where possible to cut personal costs. This computer has been
running now for just over a year with no Diskeeper installed. I
installed Auslogics Disk Defrag after about a month. So far I have not
had to defrag either partition, they are currently 2 and 3% fragmented.
This computer is used extensively every day for email, newsgroups,
downloading and uploading and general office work of image
manipulation, word processing and dtp. Synchronising every 2 or 3 days
usually indicates about 150 file deletions, 200 file changes and about
300 new files. I always use Ccleaner before every sync or backup to
rid the system of junk and temporary files.

I would have thought that there would be much more fragmentation than I
have got. I am pleased I have saved by not buying Diskeeper again and
it also saves the hard disk constantly churning away and (wearing out
quicker?), Auslogics will be used instead, probably soon, but once in
a year is pretty negligible.

Am I experiencing normal fragmented percentages or was Diskeeper over
emphasising fragmentation?

Sounds like you were using their realtime-when-idle defrag scheme. I
don't use Diskeeper but suspect it might be linked to the screen saver
which activates after the host has gone idle for awhile. Unless you did
something major that would cause severe fragmentation is a very short
time, constant defragging gives you no real benefit. It does, however,
cause problems if you are trying to run incremental or differential
backups. You could pickup every crumb as it falls while cutting slices
from a loaf of bread, or you could finish your cutting and then sweep it
all up at once.

Most users way over defrag their disks because they think the frag count
has to be tiny and as close to zero as possible. They'd do much better
looking at mobos with better I/O handling and faster hard disks. While
I schedule a defrag on the same day but before the full image backup,
the scheduled tasks is configured to run after the host has been idle
for awhile and will abort if the host becomes non-idle. Although it is
scheduled for the wee morning hours, I'm still likely to be using my
host at that time. So sometime in a few months there may be a chance
that I'm not using my host and the defrag task gets to run and also
complete. About the only time that I recall manually running a defrag
and letting it complete was when I needed to reduce the size of the
partition.

On Sun, 11 Sep 2011 01:11:14 -0500, VanguardLH wrote:

Warning: Do NOT run an incremental or differential image backup on the
same day you defrag your disk(s). You'll end up with huge backups
because all the physical relocation of the files. Incrementals and
differentials are used to create small backups and you defeat that
purpose if you defrag and then run these backups. Schedule the defrag
to run and complete on the same day and before you run a full backup,
and do not defrag on the days you run the incremental or differential
backups. Hence it is unwise to use on-the-fly defraggers that will
defrag when your computer goes idle or to configure boot-time defraggers
to run on every boot.

That surprises me. I don't see why defragging (which really isn't
necessary in the first place) would have any effect on incremental or
differential backups.

--

Char Jackson

On Sun, 11 Sep 2011 21:17:17 -0500, VanguardLH wrote:

About the only time that I recall manually running a defrag
and letting it complete was when I needed to reduce the size of the
partition.

I remember running into that situation in the late 90's, but then
Partition Magic and its cousins came along, completely eliminating the
defrag as a separate step in reducing a partition's size. Those were
the days! For the past 10 years or so, most partition managers
silently take care of that for you.

--

Char Jackson

Char Jackson wrote:
On Sun, 11 Sep 2011 21:17:17 -0500, VanguardLH wrote:

About the only time that I recall manually running a defrag
and letting it complete was when I needed to reduce the size of the
partition.

I remember running into that situation in the late 90's, but then
Partition Magic and its cousins came along, completely eliminating the
defrag as a separate step in reducing a partition's size. Those were
the days! For the past 10 years or so, most partition managers
silently take care of that for you.


Strictly speaking, defragmentation doesn't have to "move files to the
left" in the fragmentation graph. That's part of the many and varied
policies on "file positioning".

The Sysinternals "contig" program provides a basic defrag function.
It does that, by finding the first N sectors of contiguous space
big enough to hold the file you want defragged, and moving it there.
That causes files to be sprayed all over the disk (I've tried it).
But it does achieve the objective, that when you query the moved
file, it's all in one piece.

A real defragmenter, has additional policies, policies that require
moving other files out of the way, to meet the policy objective.

On the Windows defragmenter, it may choose to move the prefetch
files down near the "left end". On some of the other third party
defragmenters, they do things like put the small files on the left
and the big files on the right. There are as many variations on
file positioning, as there are days of the week.

I feel all the more lucky now, when I wanted to shrink my
laptop Windows 7 C: partition from 320GB to 30GB, that the
Raxco PerfectDisk trial I used for the purpose VanguardLH
suggests, actually did the job. I guess there wasn't
anything in System Volume Information at the time, to
stand in the way. At the time, I was testing the Windows
built in partition shrink function, which only shrinks a partition
to about 51% of original size, requiring repeated shrinks and
defrags to get the job done.

Paul

mick wrote:

I have had numerous computers running XP, Vista and Win 7. On the XP
and Vista systems I used Diskeeper for defragging continually in the
background. I cannot say whether Diskeepers' claims of using their
software gives faster performance or not as I did not experience any
slowness of any computers.

What I have done on the Win 7 computer is to set it up differently. I
have partitioned the hard drive with 100GB for the operating system and
programs, and 500GB for personal files, at the moment there is 48 and
350GB of files on these partitions. I have also opted to use free
software where possible to cut personal costs. This computer has been
running now for just over a year with no Diskeeper installed. I
installed Auslogics Disk Defrag after about a month. So far I have not
had to defrag either partition, they are currently 2 and 3% fragmented.
This computer is used extensively every day for email, newsgroups,
downloading and uploading and general office work of image
manipulation, word processing and dtp. Synchronising every 2 or 3 days
usually indicates about 150 file deletions, 200 file changes and about
300 new files. I always use Ccleaner before every sync or backup to
rid the system of junk and temporary files.

I would have thought that there would be much more fragmentation than I
have got. I am pleased I have saved by not buying Diskeeper again and
it also saves the hard disk constantly churning away and (wearing out
quicker?), Auslogics will be used instead, probably soon, but once in
a year is pretty negligible.

Am I experiencing normal fragmented percentages or was Diskeeper over
emphasising fragmentation?

Sounds like you were using their realtime-when-idle defrag scheme. I
don't use Diskeeper but suspect it might be linked to the screen saver
which activates after the host has gone idle for awhile.

That is correct.


Most users way over defrag their disks because they think the frag count
has to be tiny and as close to zero as possible.

I always thought that too. What I noticed with Diskeeper is that it
never reduced the total fragmentation to below 1%

--
mick

"VanguardLH" wrote in message ...
Jeff Layman wrote:

Probably a silly question, but would it be possible to use a linux live
CD to defrag the hard disk - including the SVI file - if such a defrag
utility existed on the CD?

The act of moving the SVI folder results in losing the System Restore
points. So instead of defragging the SVI folder, just turn off SR, lose
the restore points, and reenable SR. Since you'll lose the restore
points if you move this folder, why bother moving it? Just delete
what's inside to recover the disk space.


Not sure if that's just an effect of defrag on a 64-bit OS or not, but I used Auslogics on WinXP and Win7 (both 32-bit)
to test that, and am happy to report that the SVI was defragged (showing no fragmented files) and that all my SR points
are still showing, and the ones from two days ago still worked. Now, whether or not Auslogics moves the folder or not, I
don't know, but it did move the fragments in (of?) the folder to make the files contiguous. The only files that show up
as fragmented on the entire drive after completion are a couple of files in my temp folder, and one of them is the
defrag log file.
--
SC Tom

Char Jackson wrote:

VanguardLH wrote:

About the only time that I recall manually running a defrag and
letting it complete was when I needed to reduce the size of the
partition.

I remember running into that situation in the late 90's, but then
Partition Magic and its cousins came along, completely eliminating the
defrag as a separate step in reducing a partition's size. Those were
the days! For the past 10 years or so, most partition managers
silently take care of that for you.

I have seen info about how to disable the Journal feature of NTFS but
running "fsutil usn deletejournal /d C:" so I'm sure a defragger or
partition manager could do this, too. Apparently any app (with admin
privileges) can enable or disable the NTFS journal (as noted at
http://www.microsoft.com/msj/0999/journal/journal.aspx by using the
DeviceIoControl function). That would eliminate the otherwise unmovable
$LOGFILE files. I don't see that a reboot (restart of the OS) is
required after deleting/disabled the USN journal
(http://technet.microsoft.com/en-us/l...42(WS.10).aspx) but it
does mean that you're tossing some state info afforded by NTFS. "fsutil
usn queryjournal c:" will tell you some info but I doubt it's of any
value to you and it doesn't specify if the $LOGFILE is contiguous or
sliced across the partition.

My guess is that those running defraggers and partition managers happen
to be logged under an admin-level account so those apps could disable
journaling. But can you guarantee that all defraggers and partition
managers properly nullify features in NTFS so not to cause corruption?
I have seen users complain about boot-time defraggers and partition
managers that left them with a Stop 7B (inaccessible device) due to NTFS
corruption that left their host unusable because they cannot boot from
their hard disk.

Also, as Paul points out, resizing a partition has that utility moving
files at the end of partition to somewhere below the proposed end of the
new partition size. That's not the same as defragging a partition. Do
you know that defraggers and partition managers disable the NTFS journal
so their boot-time operation won't have any exclusively locked system
files to contend with? I've seen several users of various defraggers
complain that $UsnJrnl is huge and won't defrag. If the defragger could
simply delete it then why didn't the defragger do it? If there were no
consequence of deleting the change journal, why didn't the defragger
just go ahead and do it rather than make the user run the fsutil program
to do it manually?

From what I've seen so far, safe defraggers use the ioctrl call in
Windows but that won't let you defrag the usn journal because it is
locked for exclusive access. I've also seen it claimed that deleting
the usn journal causes no problems. It could have records of changes
made months ago on old files and is really a recovery mechanism (but
then I'm not an expert on NTFS). The file can get huge and waste a lot
of disc space (I think there is a registry setting on its max size) and
this is a rollover logfile: old entries get pushed out as new ones get
pushed in. However, I've seen it mentioned with use of the replication
service which could affect operation of other services (e.g., servers
hosting the fault-tolerant DFS for root or child node replicas); see
http://www.techrepublic.com/article/...ervice/5053993.
So there appears some problems with just blowing away the usn journal
(but may be a concern mostly just for server versions of Windows).

Although you can delete the usn journal using fsutil, it appears that it
just momentary. It is an auto-created file so more changes means the
file gets created to record those changes. Maybe that's why you must do
a boot-time defrag to delete the journal along with keeping the OS
somewhat quiescent to prevent generating a journal file (or limit its
initial occupation to the first few unused clusters in the file system
as it begins to grow).

By the way, I have used partition managers that were updated in the last
decade and still ran into errors during their resize operation. Alas,
they provide so little info on why they failed to perform the resize
that the user doesn't know how to resolve the problem. Also, despite
the "silent" (requiring reboot) pseudo-defrag (move files away from end
of partition to get them under the boundary for the new end of
partition), users have ended up with unusable partitions due to NTFS
corruption. Just because a partition manager makes it look easy doesn't
mean resizing isn't without its gotchas. The user easily makes their
choices, the program does it thing (perhaps requiring a reboot), and
then there's a failure which sometimes is not recoverable or causes lots
of problems thereafter. If you're lucky when the partition manager
fails to complete the resize, nothing got changed and you're still in
business or you lost little and what you lost wasn't critical.
http://www.google.com/search?q=%2Bpa...ize+error+fail. It
happens and with partition managers created/updated in the last decade.

While I mentioned doing my own defrag before a partition resize (since
that'll tell me if there might be problems due to unmovable files), I
failed to mention that I also save an image backup unless there's
nothing critical in the partition(s) or it's of little value or it can
be recreated. Resizing isn't the ever-safe operation you make it out to
be just because the utilities make it easy to make changes. Easy to
select doesn't make the operation safe or provide complete or even
partitial recovery after a failure. I consider partition resizing a
hazardous operation.

Char Jackson wrote:

VanguardLH wrote:

Warning: Do NOT run an incremental or differential image backup on
the same day you defrag your disk(s). You'll end up with huge
backups because all the physical relocation of the files.
Incrementals and differentials are used to create small backups and
you defeat that purpose if you defrag and then run these backups.
Schedule the defrag to run and complete on the same day and before
you run a full backup, and do not defrag on the days you run the
incremental or differential backups. Hence it is unwise to use
on-the-fly defraggers that will defrag when your computer goes idle
or to configure boot-time defraggers to run on every boot.

That surprises me. I don't see why defragging (which really isn't
necessary in the first place) would have any effect on incremental or
differential backups.

I'm not talking about logical file backups (where the file is read as a
long string through the file system, a new backup file is created with
its content, and the file gets restored through the file system which is
very similar to you running the 'copy' command in a command shell).
That's like what you get with the NT backup or Xcopy programs. It's
probably been 12-15 years since I did the old logical file backup method
ever since I went to paid/free imaging backup solutions. Since I
haven't done it that way for so long, it's not the default scheme I
think about. A defrag doesn't change the content of the file (other
than perhaps what's in its slack space but which is not normally
accessible via typical file I/O calls) so a backup method that merely
uses normal file I/O calls to read the file, store it, and then restore
it will result in the same file content wherever its clusters happen to
be inside the partition.

Sorry, but I failed to mention that I was talking about *image* backups
(and also not physical sector-by-sector exact image backups). Due to
the physical relocation of files caused by a defrag, all those moved
files become eligible for inclusion in an incremental or differential
*image* backup. Defragmentation changes the file locations on the disk,
and image backups reflect those changes. Say you do a full image backup
one day and the next day you do an incremental image backup but in
between you ran a defrag. Your incremental image defrag will be far
larger than expected (as you remember only moving the files and not
modifying their content) and could be nearly as large as your full image
backup.

If you read the FAQ or support pages for image backup programs, they'll
[should] tell you that a defrag will affect incremental and differential
backup size.

On Mon, 12 Sep 2011 10:44:57 -0500, VanguardLH wrote:

Char Jackson wrote:

VanguardLH wrote:

About the only time that I recall manually running a defrag and
letting it complete was when I needed to reduce the size of the
partition.

I remember running into that situation in the late 90's, but then
Partition Magic and its cousins came along, completely eliminating the
defrag as a separate step in reducing a partition's size. Those were
the days! For the past 10 years or so, most partition managers
silently take care of that for you.

I have seen info about how to disable the Journal feature of NTFS but
snipped unrelated info

My guess is that those running defraggers and partition managers happen
to be logged under an admin-level account so those apps could disable
journaling. But can you guarantee that all defraggers and partition
managers properly nullify features in NTFS so not to cause corruption?

I wasn't claiming to guarantee anything. I was simply saying in my
experience the two-step "defrag then resize" operation was obsolete. I
haven't run into a partition manager that required it since the late
1990's. Then again, most of my experience in the last 20 years has
been with Partition Magic until about 2002/2003 (?) and with Acronis
Disk Director since then, but there have been a few others used along
the way when I didn't have access to my primary tools. None required
me to use a separate tool to move data before allowing me to resize a
partition.

I have seen users complain about boot-time defraggers and partition
managers that left them with a Stop 7B (inaccessible device) due to NTFS
corruption that left their host unusable because they cannot boot from
their hard disk.

Users complain about a lot of things that won't happen to most people.
The Internet is a big place. We don't know the details of those
anecdotes.

Also, as Paul points out, resizing a partition has that utility moving
files at the end of partition to somewhere below the proposed end of the
new partition size.

I thought it was me who pointed that out, but maybe I just silently
agreed and moved on.

That's not the same as defragging a partition.

Actually, it has very little to do with defragging, and maybe nothing
at all. It's simply a process of moving data from one place to
another. No attempt needs to be made to defrag anything.

Do
you know that defraggers and partition managers disable the NTFS journal
so their boot-time operation won't have any exclusively locked system
files to contend with?

Don't know, don't care, never had a problem and don't expect to. Any
decent partition manager will know how to take care of data that would
otherwise fall outside of the newly proposed partition boundaries. If
it can't do that, dump it ASAP and get another (better) tool. There
are many to choose from.

snipped multiple paragraphs of unrelated info

Resizing isn't the ever-safe operation you make it out to
be just because the utilities make it easy to make changes. Easy to
select doesn't make the operation safe or provide complete or even
partitial recovery after a failure. I consider partition resizing a
hazardous operation.

I think you're putting words in my mouth.

--

Char Jackson

Char Jackson wrote:

On Mon, 12 Sep 2011 10:44:57 -0500, VanguardLH wrote:

Char Jackson wrote:

VanguardLH wrote:

About the only time that I recall manually running a defrag and
letting it complete was when I needed to reduce the size of the
partition.

I remember running into that situation in the late 90's, but then
Partition Magic and its cousins came along, completely eliminating the
defrag as a separate step in reducing a partition's size. Those were
the days! For the past 10 years or so, most partition managers
silently take care of that for you.

I have seen info about how to disable the Journal feature of NTFS but
snipped unrelated info

My guess is that those running defraggers and partition managers happen
to be logged under an admin-level account so those apps could disable
journaling. But can you guarantee that all defraggers and partition
managers properly nullify features in NTFS so not to cause corruption?

I wasn't claiming to guarantee anything. I was simply saying in my
experience the two-step "defrag then resize" operation was obsolete. I
haven't run into a partition manager that required it since the late
1990's. Then again, most of my experience in the last 20 years has
been with Partition Magic until about 2002/2003 (?) and with Acronis
Disk Director since then, but there have been a few others used along
the way when I didn't have access to my primary tools. None required
me to use a separate tool to move data before allowing me to resize a
partition.

I have seen users complain about boot-time defraggers and partition
managers that left them with a Stop 7B (inaccessible device) due to NTFS
corruption that left their host unusable because they cannot boot from
their hard disk.

Users complain about a lot of things that won't happen to most people.
The Internet is a big place. We don't know the details of those
anecdotes.

An opinion from one user regarding their personal use has some but
little value when making blanket statements. Perhaps you haven't needed
to fix computers for other users.

Actually, it has very little to do with defragging, and maybe nothing
at all. It's simply a process of moving data from one place to
another. No attempt needs to be made to defrag anything.

But moving files with a partition manager that are exclusively locked
still runs into the same problems with defraggers moving those same
files. So basically the issues with defraggers are the same as for
partition managers when moving around the files to different clusters.

Don't know, don't care, never had a problem and don't expect to.

Yes, got that, you haven't run into the problem with a non-bootable
partition after using a boot-time defragger or a partition manager that
reboots and then proceeds to move about the system files along with
those within the NTFS file system. By the way, you (and I) really
didn't mention if you/me use NTFS. I use it exclusively except for USB
flash drives. You haven't run into a non-bootable partition after a
resize, I have, so the voting on personal experience is even; however,
other users do encounter problems so to dismiss them simply because you
haven't encountered them doesn't make them non-existent or impossible
and not even improbable.

Any decent partition manager will know how to take care of data that
would otherwise fall outside of the newly proposed partition
boundaries. If it can't do that, dump it ASAP and get another
(better) tool. There are many to choose from.

If defraggers cannot move around the unmovable files, just how is any
partition manager going to do it? Some defraggers can do it (at boot
time) but some users HAVE encountered problems thereafter. I'm not
saying it's a prevalent problem but it does happen. When it happens to
you (or you're stuck having to repair someone else's computer where it
happens), it suddenly isn't such rare a problem.

It hasn't happened to you. Good for you. It has happened to me. When
you get burned, you figure out how to prevent it, not just wait to see
if it happens again. Pain is an excellent learning motivator.

Char Jackson wrote:


Any decent partition manager will know how to take care of data that would
otherwise fall outside of the newly proposed partition boundaries.

That was my point, of what I was doing with the Windows 7 built-in "shrink"
function. Unlike the third party competition, Microsoft didn't bother
moving everything that could possibly be moved (which is why is stops at 51%).
I found a defragmenter with the required capability and file movement policies
(ability to move metadata files to the left a bit). And by using multiple passes
of the two tools (defragmenter + Microsoft "shrink" function), I got the job done.

I presume a purpose built partition manager tool, would have to do
something similar, have the ability to move metadata around. Otherwise
it would get stuck like the Microsoft implementation.

In the picture here, I think it was the two block "grey thing" at around the 50% mark,
that causes the problem. I didn't keep any screenshots of what mine was doing.
The defragmenter won't push it all the way to the left either, so they're
following some rule about its position.

http://img841.imageshack.us/img841/7...01111431pm.png

*******

I found a utility which might have some fun value. It was mentioned
in the Wikipedia article on NTFS. (This is for if you don't own a
commercial defragmenter, and want to know where things are. It also
demonstrates fragmentation, in that fragmented files have more than
one group of sectors listed.)

http://support.microsoft.com/kb/253066/

http://download.microsoft.com/downlo...s/oem3sr2s.zip

It's got a copy of nti.exe in it. If I run that on a NTFS (data) partition here
on my WinXP machine, it gave about a 4MB dump, similar to this. The
"two grey blocks" in the middle, could be $MftMirr.

File 0
Master File Table ($Mft)
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 32-25087 (0x20-0x61ff)
$BITMAP (nonresident)
logical sectors 16-23 (0x10-0x17)

File 1
Master File Table Mirror ($MftMirr)
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 428670424-428670431 (0x198cfdd8-0x198cfddf)

....

File 12441
\27a368f93559180042cd5c7058f99841\Setup\custsat_x8 6.dll
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 421623008-421623087 (0x192174e0-0x1921752f)

So if nothing else, you can use it as a file listing tool :-)

The volume I tested that on, is 857340855 sectors, and it's only
about half full. One peculiarity, is not every "File" is
listed, and the file numbering skips at one point. The
volume has 11097 files and 1311 directories or 12408 objects.
So the count is roughly in the right ballpark. Still, pretty
damn good for a program which is only 21,744 bytes in size.
Good things come in small packages :-)

Paul