Disk Uses More Space Than Size of Files

W wrote:
"VanguardLH" wrote in message
...
"VanguardLH" wrote ...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft


http://www.symantec.com/connect/arti...e-data-streams

I found a better utility for scanning for ADS: NirSoft
AlternateStreamView. Strangely, this utility does not agree on all of the
results with the ADS Scanner you mentioned.

Some surprising things I found:

1) Dropbox is using the ADS feature actively, and many dropbox files have up
to 4096 bytes of ADS information attached to them.

2) The AlternateStreamView shows an additional field of
"StreamAllocatedSize". In my boot partition I had a few files where the
actual ADS stream was about 1K but the ADS Allocated Size was about 65K.
Does anyone know if the "allocated size" represents actual disk space in
use?

Most of these files that AlternateStreamView reported large allocation sizes
on were not even seen by ADS Scanner.


Amazingly, it notes that concept here. That StreamAllocatedSize is set aside
for the stream for some reason. I don't know if this idea is intended for
sparse storage, or what they were thinking. You would have thought, if they
were going to waste space, they'd round to the nearest 4K or whatever.

http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

For a stream to work, it would have to have a provision for extending the
storage space later (leading to fragmentation). Pre-allocating a space,
isn't of much use if you don't "guess right" on the size to set aside.
So I'm missing what the big gain is, in having such a feature.

Paul

W wrote:

"VanguardLH" wrote ...

W wrote:

The backup files are being made by Acronis True Image.

Ahhh. I use that, too. You didn't happen to use their "Try & Decide"
feature, did you? T&D demands the use of the Acronis Secure Zone (ASZ),
a separate partition on a hard disk formatted as FAT32 but uses a
non-standard partition type number in the MBR.

No, I did not use Try and Decide.

So the partition in question is not the Acronis Secure Zone, correct?

W wrote:

I found a better utility for scanning for ADS: NirSoft
AlternateStreamView. Strangely, this utility does not agree on all of the
results with the ADS Scanner you mentioned.

Some surprising things I found:

1) Dropbox is using the ADS feature actively, and many dropbox files have up
to 4096 bytes of ADS information attached to them.

2) The AlternateStreamView shows an additional field of
"StreamAllocatedSize". In my boot partition I had a few files where the
actual ADS stream was about 1K but the ADS Allocated Size was about 65K.
Does anyone know if the "allocated size" represents actual disk space in
use?

Most of these files that AlternateStreamView reported large allocation sizes
on were not even seen by ADS Scanner.

Forgot about that one. I've used many Nirsoft tools and found many
anti-virus programs detect them as PUPs (probably unwanted programs) and
won't change their status upon request so I had to whitelist them or add
to an exclude list in their settings so the scanner doesn't generate
false alerts again.

ADS Scanner is an old tool. Many ADS tools are old, like the Stream
Explorer from Rekenwonder. Also, the authors may not know as much about
NTFS as does Nirsoft.

http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

"Reserved" doesn't have a consistent definition. Sometimes it means the
space is locked out from use by anything else (i.e., it is actually
allocated but possibly not completely yet consumed), like reserving a
table so no one else can sit at it whether you're there or not.
Sometimes it's more of a maximum size attribute in that it is the
maximum space an object can consume but that space could be used by
something else. The space is reserved but not locked out so something
else could use it and then the caller that wanted the reserved space
won't have it. The file system tries to put other files outside the
reserved space until it runs out of other free space and has to use up
the reserved space.

However, it seems we've focused on ADS yet that is not where gobs of
disk space are getting consumed. Even if every backup image file in
that partition had a 65KB sized stream attached to it, I'm assuming
there aren't thousands of backup files in this partition so something
else is eating up that 350GB of disk space. I mentioned ADS to have you
check if there were HUGE streams attached to any files, not tiny ones.
It's when HUGE streams get secretely attached to files that users are at
a loss as to why the file system complains there is no space left even
for creating a small file.

In fact, since the ADS size is not accounted for in Windows Explorer,
you won't see that disk space as a loss in capacity for a partition.
You said:

- 1 TB NTFS partition.
- Windows Explorer's properties on the drive show only 70GB free.
- Your calculation (which should be on allocated disk size, not actual
bytes for the file) on file space says there should be 350GB free.

If there were some huge streams attached to files that gobbled up 280GB,
you wouldn't see that in Windows Explorer. Despite those huge streams,
Windows Explorer would still show 350GB free. It doesn't include stream
consumption. It only shows space used by the primary (default) streams.
When ADS is the culprit, users complain that Windows Explorer shows far
more than enough free space left on a drive but the users cannot create
even a 1KB (4KB for its cluster [if using defaults]) sized file. If you
don't find HUGE streams attached to files the total of which represents
the hidden lost capacity then we're off focusing on minutia.

Is this a default formatted partition where cluster sizes are 4096 bytes
in size? Or did you specify a different allocation unit (cluster) size
during format, like 64KB? Run the following command in a console
(command shell):

fsutil fsinfo ntfsinfo u:

where u: is your backup partition/drive. That will tell you what is the
cluster size.

"VanguardLH" wrote in message
...
W wrote:

"VanguardLH" wrote ...

W wrote:

The backup files are being made by Acronis True Image.

Ahhh. I use that, too. You didn't happen to use their "Try & Decide"
feature, did you? T&D demands the use of the Acronis Secure Zone
(ASZ),
a separate partition on a hard disk formatted as FAT32 but uses a
non-standard partition type number in the MBR.

No, I did not use Try and Decide.

So the partition in question is not the Acronis Secure Zone, correct?

Correct, and further I do not use any Acronis Secure Zone.

--
W

Originally Posted by W[_2_]

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?

--
W

Run Disk Defragmenter please.