Disk Uses More Space Than Size of Files

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?

--
W

W wrote:
I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?


The mystery area is probably System Volume Information.

Make sure System Restore is not "tracking" your backup drive.
WinXP tends to turn that on by default, each time a new drive
is connected. I have System Restore turned off on WinXP now.
So that's no longer an issue. Turning off System Restore,
does not remove SVI folder, neither does it disable
VSS service (which is used when backing up a partition
with a modern backup tool).

I wish there was an easy way to visualize what's in there,
so I could do more testing on this.

You could try ShadowExplorer, which is supposed to make
visible things that are being shadowed (and hidden) inside
SVI. But ShadowExplorer doesn't show you everything
stored in SVI. In WinXP, you see lots of RPxx folders
for restore points, as a major contributor to bloat.
Turning off System Restore, should make those go away
(on all your partitions, not just the one you're concerned
about right now). It really depends on whether you consider
System Restore to be worthwhile, as to whether it should
stay turned on. When you do Windows Update, that is one time
it's nice to have it turned on, just in case.

I boot a Linux LiveCD for a look inside System Volume Information.
On Vista/Win7/Win8, it's possible to trash the OS by screwing
around in that folder. So take precautions before becoming
too adventurous. I had to restore my laptop from a backup,
after one of my little experiments ran amok. And using the
Linux LiveCD was all part of that (causing the problem) :-)

One of my Linux LiveCD collection, the Knoppix 5.3.1 DVD,
mounts all partitions read-only. And while I don't use
that disc regularly any more, that's the only one I trust
not to trash stuff. It's like a gun with a safety -
you can still turn off the safety on that LiveCD environment,
and trash stuff. But it stops you, if you're a noob at it.
To go read/write, takes an extra step.

Paul

"Paul" wrote in message
...
W wrote:
I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of
all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up
the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in
size.
So there are no small files on the partition that would waste empty
sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS?
What
tools might help me to explore this further?


The mystery area is probably System Volume Information.

Make sure System Restore is not "tracking" your backup drive.
WinXP tends to turn that on by default, each time a new drive
is connected. I have System Restore turned off on WinXP now.
So that's no longer an issue. Turning off System Restore,
does not remove SVI folder, neither does it disable
VSS service (which is used when backing up a partition
with a modern backup tool).

I wish there was an easy way to visualize what's in there,
so I could do more testing on this.

You could try ShadowExplorer, which is supposed to make
visible things that are being shadowed (and hidden) inside
SVI. But ShadowExplorer doesn't show you everything
stored in SVI. In WinXP, you see lots of RPxx folders
for restore points, as a major contributor to bloat.
Turning off System Restore, should make those go away
(on all your partitions, not just the one you're concerned
about right now). It really depends on whether you consider
System Restore to be worthwhile, as to whether it should
stay turned on. When you do Windows Update, that is one time
it's nice to have it turned on, just in case.

I boot a Linux LiveCD for a look inside System Volume Information.
On Vista/Win7/Win8, it's possible to trash the OS by screwing
around in that folder. So take precautions before becoming
too adventurous. I had to restore my laptop from a backup,
after one of my little experiments ran amok. And using the
Linux LiveCD was all part of that (causing the problem) :-)

One of my Linux LiveCD collection, the Knoppix 5.3.1 DVD,
mounts all partitions read-only. And while I don't use
that disc regularly any more, that's the only one I trust
not to trash stuff. It's like a gun with a safety -
you can still turn off the safety on that LiveCD environment,
and trash stuff. But it stops you, if you're a noob at it.
To go read/write, takes an extra step.

I turned off System Volume Information on the backup drive, just to be safe,
but there was nothing inside that folder.

I don't understand why you wouldn't just give Local Administrators group
READ access to System Volume Information. I have done this for a long time
on many machines and have never seen any side effect.

--
W

W wrote:

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?

Did you include hidden-marked files? Did you configure Windows Explorer
to show hidden-marked files?

Did you include special and OS files not shown by Windows Explorer? Did
you include the pagefile and hibernate files? You didn't explicitly
state this was a non-OS partition. Even if it is not an OS partition,
but if the partition is on a different hard disk, often users will
spread the pagefile across multiple hard disks so reads can be queued at
the same time to the pagefile along with those to the OS partition.
Maybe the pagefile portion in the problematic partition, if enabled, is
set way too huge.

Is that partition only for use by your backup program? And that backup
is which one? Some backup programs provide for snapshots. They
deliberately hide those snapshots from the file system, so what you see
in Windows Explorer won't reflect the space consumed by the snapshots.
Then there are other "restore" utilities, like Comodo's Time Machine
(don't ever try using this betaware) or DeepFreeze, that take snaphots
of the file system and hide them from the OS. If you are using
virtualized disk I/O utilities to provide a safe environment for testing
unknown or untrusted software (just the disk is virtualized and all
other hardware is real) instead of an always slower virtual machine,
like Returnil, they hide their disk space used by their replacement file
I/O handler to virtualize disk I/O. Some folks have this setup to use
half of the available free space. It isn't in use when you are not in
"safe mode" (when virtualized disk is active) so that space is
available; however, once you go into safe (virtualized disk) mode then
that space gets used. While virtualize disk I/O is active, that space
can get consumed. When you reboot, all changes to the virtual disk are
discarded and you're back to using the real disk I/O. The paid version
lets you incorporate all changes to the virtual disk onto the real disk.
Are you using any kind of this software that might be reserving disk
space while it is active? Returnil, for example, can be set to always
boot into safe mode so the only way out is to have admin rights to
change its config to not boot into safe mode and to be back to real disk
mode. SteadyState and other products do similar disk virtualizing so
all changes can be discarded but you have direct access to all other
hardware so, for example, your video game runs at full speed using the
real video card instead of an emulated one used inside virtual machines.

If you are doing regular/daily backups then you really don't need System
Restore. Disable System Restore (at least on that partition) which will
delete all restore points for that partition. Restore points won't
include you backup files, anyway, so there's no point in enabling System
Restore on that partition.

"VanguardLH" wrote in message
...
W wrote:

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of
all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up
the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in
size.
So there are no small files on the partition that would waste empty
sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS?
What
tools might help me to explore this further?

Did you include hidden-marked files? Did you configure Windows Explorer
to show hidden-marked files?

Did you include special and OS files not shown by Windows Explorer? Did
you include the pagefile and hibernate files? You didn't explicitly
state this was a non-OS partition. Even if it is not an OS partition,
but if the partition is on a different hard disk, often users will
spread the pagefile across multiple hard disks so reads can be queued at
the same time to the pagefile along with those to the OS partition.
Maybe the pagefile portion in the problematic partition, if enabled, is
set way too huge.

Is that partition only for use by your backup program? And that backup
is which one? Some backup programs provide for snapshots. They
deliberately hide those snapshots from the file system, so what you see
in Windows Explorer won't reflect the space consumed by the snapshots.
Then there are other "restore" utilities, like Comodo's Time Machine
(don't ever try using this betaware) or DeepFreeze, that take snaphots
of the file system and hide them from the OS. If you are using
virtualized disk I/O utilities to provide a safe environment for testing
unknown or untrusted software (just the disk is virtualized and all
other hardware is real) instead of an always slower virtual machine,
like Returnil, they hide their disk space used by their replacement file
I/O handler to virtualize disk I/O. Some folks have this setup to use
half of the available free space. It isn't in use when you are not in
"safe mode" (when virtualized disk is active) so that space is
available; however, once you go into safe (virtualized disk) mode then
that space gets used. While virtualize disk I/O is active, that space
can get consumed. When you reboot, all changes to the virtual disk are
discarded and you're back to using the real disk I/O. The paid version
lets you incorporate all changes to the virtual disk onto the real disk.
Are you using any kind of this software that might be reserving disk
space while it is active? Returnil, for example, can be set to always
boot into safe mode so the only way out is to have admin rights to
change its config to not boot into safe mode and to be back to real disk
mode. SteadyState and other products do similar disk virtualizing so
all changes can be discarded but you have direct access to all other
hardware so, for example, your video game runs at full speed using the
real video card instead of an emulated one used inside virtual machines.

If you are doing regular/daily backups then you really don't need System
Restore. Disable System Restore (at least on that partition) which will
delete all restore points for that partition. Restore points won't
include you backup files, anyway, so there's no point in enabling System
Restore on that partition.

I always configure Windows Explorer to show hidden and system files. It's
one of my first setup steps. In any case, there were no large hidden or
system files. But having an application that would search the entire
partition and then sort files from largest to smallest would not be a bad
utility to have.

The drive and partition in question only holds backup files. So there is no
paging file or hibernation. It's my U partition on a separate SATA drive.
This is a "real" partition and I'm not running this OS as a virtual machine.

I did turn off System Restore.

--
W

"VanguardLH" wrote ...

And that backup is which one?

Still waiting on that question.

In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft

http://www.symantec.com/connect/arti...e-data-streams

While there are legitimate uses of ADS, it can also be misused.

http://www.irongeek.com/i.php?page=security/altds

That shows how using just the simple 'echo' console command that you can
add a huge file onto a text file. By redirecting stdout to the target
file but specifying a name for an ADS (the part after the colon in the
filename), all that stdout goes into the ADS. While that article looks
old, ADS is still a feature of NTFS.

You didn't say if the backup partition on the external drive is
formatted using FAT32, NTFS, exFAT, or some other file system.

You can find ADS utilities to expose the multiple streams (the blank or
no name one is the primary one or the one you normally consider the
file itself). http://www.rekenwonder.com/streamexplorer.htm is one such
utility but there are probably lots of these. I've used this one in the
past but obviously it's mostly to reveal there is an ADS on file that
you select rather than scan all your files to find which ones have one,
or more, ADS attached to them. You might want to ask in the
alt.comp.freeware newsgroup (get ready to ignore lots of noise) on what
is a good ADS explorer tool. As I recall, there was one ran from the
command line that would strip all ADS from the specified files but then
you lose any meta data they stored, like a thumbnail image. I once had
such a command-line scanner tool so I know that you'll find lots of
files that have an ADS for them but often it's trivial meta-data.

I do remember that some backup programs use the ADS to keep track of
their versioning history. Don't remember which one but recall one that
used the ADS to record if a file had already been backed up and the hash
value for the file at that time. For a subsequent incremental backup
job, it could use that meta-data to determine if it could skip an
unchanged file. The archive file attribute is not a reliable means of
determining if a file has changed or not so meta-data was used to keep
track if a file (in its current state) had already been backed up. I
even recall an anti-virus program doing that (I think it was Kaspersky)
so it could shorten its on-demand scans. If the file hadn't changed
since the last scan and it was included in the scan, info stored as
meta-data in an ADS on the file, then the AV's scan could skip that file
to eliminate wasting time rechecking a file that had already been
checked before, didn't change, so it doesn't need to be checked again.
If you used an ADS scanner to strip them from files, the AV program
would have to scan all files again.

http://www.softpedia.com/progScreens...ot-134764.html

That's an example of a scanner so you don't have to manually select
files in an explorer tool to see if they have an ADS and what is its
size. I've never used ADS Scanner. Just remember there are also
legimate uses of ADS so stripping them off means losing data which could
affect functionality within the OS or an app that handles the file.

SysInternals has their 'streams' utility to scan from a command prompt
for ADS on files. Without the -d switch, it'll list which files were
found to have an ADS. With the -d switch, it would delete the ADS file
on every matching file to the filespec you gave it. Again, you could
end up deleting more than you want, so look before deleting.

"VanguardLH" wrote in message
...
"VanguardLH" wrote ...

And that backup is which one?

Still waiting on that question.

The backup files are being made by Acronis True Image.


In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft


http://www.symantec.com/connect/arti...e-data-streams

While there are legitimate uses of ADS, it can also be misused.

http://www.irongeek.com/i.php?page=security/altds

That shows how using just the simple 'echo' console command that you can
add a huge file onto a text file. By redirecting stdout to the target
file but specifying a name for an ADS (the part after the colon in the
filename), all that stdout goes into the ADS. While that article looks
old, ADS is still a feature of NTFS.

So is there a utility that will give you a view of all space being used by a
file system tree, including the ADS?


You didn't say if the backup partition on the external drive is
formatted using FAT32, NTFS, exFAT, or some other file system.

Everything is NTFS.


You can find ADS utilities to expose the multiple streams (the blank or
no name one is the primary one or the one you normally consider the
file itself). http://www.rekenwonder.com/streamexplorer.htm is one such
utility but there are probably lots of these. I've used this one in the
past but obviously it's mostly to reveal there is an ADS on file that
you select rather than scan all your files to find which ones have one,
or more, ADS attached to them. You might want to ask in the
alt.comp.freeware newsgroup (get ready to ignore lots of noise) on what
is a good ADS explorer tool. As I recall, there was one ran from the
command line that would strip all ADS from the specified files but then
you lose any meta data they stored, like a thumbnail image. I once had
such a command-line scanner tool so I know that you'll find lots of
files that have an ADS for them but often it's trivial meta-data.

Obviously looking file by file doesn't scale well to a large number of
files. You want a utility that will look across whole subtrees and provide
summaries that let you dig into specific areas for further research.


I do remember that some backup programs use the ADS to keep track of
their versioning history. Don't remember which one but recall one that
used the ADS to record if a file had already been backed up and the hash
value for the file at that time. For a subsequent incremental backup
job, it could use that meta-data to determine if it could skip an
unchanged file. The archive file attribute is not a reliable means of
determining if a file has changed or not so meta-data was used to keep
track if a file (in its current state) had already been backed up. I
even recall an anti-virus program doing that (I think it was Kaspersky)
so it could shorten its on-demand scans. If the file hadn't changed
since the last scan and it was included in the scan, info stored as
meta-data in an ADS on the file, then the AV's scan could skip that file
to eliminate wasting time rechecking a file that had already been
checked before, didn't change, so it doesn't need to be checked again.
If you used an ADS scanner to strip them from files, the AV program
would have to scan all files again.

I went ahead and installed ADS Scanner and am running it. It's an extremely
primitive tool and somewhat buggy, but at this point it is a start, thanks.

--
W

W wrote:

The backup files are being made by Acronis True Image.

Ahhh. I use that, too. You didn't happen to use their "Try & Decide"
feature, did you? T&D demands the use of the Acronis Secure Zone (ASZ),
a separate partition on a hard disk formatted as FAT32 but uses a
non-standard partition type number in the MBR.

In fact, if you try to use T&D, you get somewhat screwed in that it
demands it usurp the bootstrap code in the MBR which obviates using the
Acronis Recovery Manager that also usurps the MBR bootstrap area. Only
1 can usurp the MBR bootstrap code at a time. If you want Acronis
Recovery Manager then you cannot use T&D. If you want to use T&D, you
have to forego availability of the Acronis Recovery Manager. Of course,
using either of these obviates using any other program that wants to
usurp the MBR bootstrap area (e.g., GAG, a multi-boot manager that
resides wholly outside any partition). In older versions, Acronis
Recovery Manager and T&D were compatible. It was in the last version,
or two, they changed it so they were mutually exclusive because they
changes T&D to want to usurp the MBR bootstrap code.

If you are saving your backup images in the ASZ and also using T&D then
both the image backups and T&D's virtual disk share the same partition.
Make sure T&D is disabled (off). If it is then maybe there is a problem
with it releasing the disk space in the ASZ partition. It could be that
the space for T&D is merely "reserved". That means if T&D is disabled
that Acronis will use that space for its backups. Reserved doesn't
necessarily mean inuse.

"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.

--
W

W wrote:
"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.


If you're dealing with NTFS, you want a copy of nfi.exe.

It too is a crude utility, but it does present all the
useful information. To be really useful, it needs post-processing
with scripts. It only works on NTFS, and there is no equivalent
for FAT32.

Still, when no other utility is available or convenient, you
use what you've got.

http://support.microsoft.com/kb/253066
http://download.microsoft.com/downlo...us/oem3sr2.zip

When you find two files that have the same set of data sectors,
those files are probably hard linked, and get double-counted
by Explorer while you're attempting to total the space. Only
the "summary pie chart" in Windows, tells you how much
space the partition is really using. Attempting to use
a hand calculator and mousing over folders in the (file) Explorer
is a waste of time. More so on Vista/Win7/Win8, as things like
hard linking aren't really used that much on something like
WinXP.

*******

There are still a few files on NTFS, that no utility
will touch. For that case, there's Linux...
Even nfi.exe doesn't list everything. Compare the
listing from Linux with Windows, for more info about
what you might be missing. It's even possible
if you use listdir in Linux and list by inode, the
fake inode will correspond to the file number
seen in nfi. But I haven't tested for that.
That's just a guess.

Paul

W wrote:

"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.

There are utilities to add-on to Windows Explorer to let users see the
streams, if any (other than the default/primary one), attached to a
file, like:

http://www.jsware.net/jsware/sviewer.php5

Haven't used it so cannot comment on its usefulness. There are probably
other shell extensions that make it convenient to check for and view
streams on folders or files using Windows Explorer.

Gets even worse in Windows 7, and later, where Microsoft decided to use
this NTFS feature to further block file access by even admin-level users
(whether they are in the Administrators group using their own account or
even if using the Administrator account). See:

http://www.jsware.net/jsware/nt6fix.php5

Their NT6 Fix utility sounds a lot like the Take Ownership utility that
I installed that appears as a context menu entry when I right-click on a
folder or file.

"VanguardLH" wrote in message
...
"VanguardLH" wrote ...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft


http://www.symantec.com/connect/arti...e-data-streams

I found a better utility for scanning for ADS: NirSoft
AlternateStreamView. Strangely, this utility does not agree on all of the
results with the ADS Scanner you mentioned.

Some surprising things I found:

1) Dropbox is using the ADS feature actively, and many dropbox files have up
to 4096 bytes of ADS information attached to them.

2) The AlternateStreamView shows an additional field of
"StreamAllocatedSize". In my boot partition I had a few files where the
actual ADS stream was about 1K but the ADS Allocated Size was about 65K.
Does anyone know if the "allocated size" represents actual disk space in
use?

Most of these files that AlternateStreamView reported large allocation sizes
on were not even seen by ADS Scanner.

--
W

W wrote:
"VanguardLH" wrote in message
...
"VanguardLH" wrote ...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft


http://www.symantec.com/connect/arti...e-data-streams

I found a better utility for scanning for ADS: NirSoft
AlternateStreamView. Strangely, this utility does not agree on all of the
results with the ADS Scanner you mentioned.

Some surprising things I found:

1) Dropbox is using the ADS feature actively, and many dropbox files have up
to 4096 bytes of ADS information attached to them.

2) The AlternateStreamView shows an additional field of
"StreamAllocatedSize". In my boot partition I had a few files where the
actual ADS stream was about 1K but the ADS Allocated Size was about 65K.
Does anyone know if the "allocated size" represents actual disk space in
use?

Most of these files that AlternateStreamView reported large allocation sizes
on were not even seen by ADS Scanner.


Amazingly, it notes that concept here. That StreamAllocatedSize is set aside
for the stream for some reason. I don't know if this idea is intended for
sparse storage, or what they were thinking. You would have thought, if they
were going to waste space, they'd round to the nearest 4K or whatever.

http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

For a stream to work, it would have to have a provision for extending the
storage space later (leading to fragmentation). Pre-allocating a space,
isn't of much use if you don't "guess right" on the size to set aside.
So I'm missing what the big gain is, in having such a feature.

Paul

W wrote:

I found a better utility for scanning for ADS: NirSoft
AlternateStreamView. Strangely, this utility does not agree on all of the
results with the ADS Scanner you mentioned.

Some surprising things I found:

1) Dropbox is using the ADS feature actively, and many dropbox files have up
to 4096 bytes of ADS information attached to them.

2) The AlternateStreamView shows an additional field of
"StreamAllocatedSize". In my boot partition I had a few files where the
actual ADS stream was about 1K but the ADS Allocated Size was about 65K.
Does anyone know if the "allocated size" represents actual disk space in
use?

Most of these files that AlternateStreamView reported large allocation sizes
on were not even seen by ADS Scanner.

Forgot about that one. I've used many Nirsoft tools and found many
anti-virus programs detect them as PUPs (probably unwanted programs) and
won't change their status upon request so I had to whitelist them or add
to an exclude list in their settings so the scanner doesn't generate
false alerts again.

ADS Scanner is an old tool. Many ADS tools are old, like the Stream
Explorer from Rekenwonder. Also, the authors may not know as much about
NTFS as does Nirsoft.

http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

"Reserved" doesn't have a consistent definition. Sometimes it means the
space is locked out from use by anything else (i.e., it is actually
allocated but possibly not completely yet consumed), like reserving a
table so no one else can sit at it whether you're there or not.
Sometimes it's more of a maximum size attribute in that it is the
maximum space an object can consume but that space could be used by
something else. The space is reserved but not locked out so something
else could use it and then the caller that wanted the reserved space
won't have it. The file system tries to put other files outside the
reserved space until it runs out of other free space and has to use up
the reserved space.

However, it seems we've focused on ADS yet that is not where gobs of
disk space are getting consumed. Even if every backup image file in
that partition had a 65KB sized stream attached to it, I'm assuming
there aren't thousands of backup files in this partition so something
else is eating up that 350GB of disk space. I mentioned ADS to have you
check if there were HUGE streams attached to any files, not tiny ones.
It's when HUGE streams get secretely attached to files that users are at
a loss as to why the file system complains there is no space left even
for creating a small file.

In fact, since the ADS size is not accounted for in Windows Explorer,
you won't see that disk space as a loss in capacity for a partition.
You said:

- 1 TB NTFS partition.
- Windows Explorer's properties on the drive show only 70GB free.
- Your calculation (which should be on allocated disk size, not actual
bytes for the file) on file space says there should be 350GB free.

If there were some huge streams attached to files that gobbled up 280GB,
you wouldn't see that in Windows Explorer. Despite those huge streams,
Windows Explorer would still show 350GB free. It doesn't include stream
consumption. It only shows space used by the primary (default) streams.
When ADS is the culprit, users complain that Windows Explorer shows far
more than enough free space left on a drive but the users cannot create
even a 1KB (4KB for its cluster [if using defaults]) sized file. If you
don't find HUGE streams attached to files the total of which represents
the hidden lost capacity then we're off focusing on minutia.

Is this a default formatted partition where cluster sizes are 4096 bytes
in size? Or did you specify a different allocation unit (cluster) size
during format, like 64KB? Run the following command in a console
(command shell):

fsutil fsinfo ntfsinfo u:

where u: is your backup partition/drive. That will tell you what is the
cluster size.

Originally Posted by W[_2_]

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?

--
W

Run Disk Defragmenter please.