A Windows XP help forum. PCbanter

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » PCbanter forum » Microsoft Windows XP » Hardware and Windows XP
Site Map Home Register Authors List Search Today's Posts Mark Forums Read Web Partners

Disk Uses More Space Than Size of Files



 
 
Thread Tools Display Modes
  #1  
Old October 20th 13, 12:13 AM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?

--
W


Ads
  #2  
Old October 20th 13, 01:51 AM posted to microsoft.public.windowsxp.hardware
Paul
external usenet poster
 
Posts: 18,281
Default Disk Uses More Space Than Size of Files

W wrote:
I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?


The mystery area is probably System Volume Information.

Make sure System Restore is not "tracking" your backup drive.
WinXP tends to turn that on by default, each time a new drive
is connected. I have System Restore turned off on WinXP now.
So that's no longer an issue. Turning off System Restore,
does not remove SVI folder, neither does it disable
VSS service (which is used when backing up a partition
with a modern backup tool).

I wish there was an easy way to visualize what's in there,
so I could do more testing on this.

You could try ShadowExplorer, which is supposed to make
visible things that are being shadowed (and hidden) inside
SVI. But ShadowExplorer doesn't show you everything
stored in SVI. In WinXP, you see lots of RPxx folders
for restore points, as a major contributor to bloat.
Turning off System Restore, should make those go away
(on all your partitions, not just the one you're concerned
about right now). It really depends on whether you consider
System Restore to be worthwhile, as to whether it should
stay turned on. When you do Windows Update, that is one time
it's nice to have it turned on, just in case.

I boot a Linux LiveCD for a look inside System Volume Information.
On Vista/Win7/Win8, it's possible to trash the OS by screwing
around in that folder. So take precautions before becoming
too adventurous. I had to restore my laptop from a backup,
after one of my little experiments ran amok. And using the
Linux LiveCD was all part of that (causing the problem) :-)

One of my Linux LiveCD collection, the Knoppix 5.3.1 DVD,
mounts all partitions read-only. And while I don't use
that disc regularly any more, that's the only one I trust
not to trash stuff. It's like a gun with a safety -
you can still turn off the safety on that LiveCD environment,
and trash stuff. But it stops you, if you're a noob at it.
To go read/write, takes an extra step.

Paul
  #3  
Old October 20th 13, 05:33 AM posted to microsoft.public.windowsxp.hardware
VanguardLH[_2_]
external usenet poster
 
Posts: 9,399
Default Disk Uses More Space Than Size of Files

W wrote:

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in size.
So there are no small files on the partition that would waste empty sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS? What
tools might help me to explore this further?


Did you include hidden-marked files? Did you configure Windows Explorer
to show hidden-marked files?

Did you include special and OS files not shown by Windows Explorer? Did
you include the pagefile and hibernate files? You didn't explicitly
state this was a non-OS partition. Even if it is not an OS partition,
but if the partition is on a different hard disk, often users will
spread the pagefile across multiple hard disks so reads can be queued at
the same time to the pagefile along with those to the OS partition.
Maybe the pagefile portion in the problematic partition, if enabled, is
set way too huge.

Is that partition only for use by your backup program? And that backup
is which one? Some backup programs provide for snapshots. They
deliberately hide those snapshots from the file system, so what you see
in Windows Explorer won't reflect the space consumed by the snapshots.
Then there are other "restore" utilities, like Comodo's Time Machine
(don't ever try using this betaware) or DeepFreeze, that take snaphots
of the file system and hide them from the OS. If you are using
virtualized disk I/O utilities to provide a safe environment for testing
unknown or untrusted software (just the disk is virtualized and all
other hardware is real) instead of an always slower virtual machine,
like Returnil, they hide their disk space used by their replacement file
I/O handler to virtualize disk I/O. Some folks have this setup to use
half of the available free space. It isn't in use when you are not in
"safe mode" (when virtualized disk is active) so that space is
available; however, once you go into safe (virtualized disk) mode then
that space gets used. While virtualize disk I/O is active, that space
can get consumed. When you reboot, all changes to the virtual disk are
discarded and you're back to using the real disk I/O. The paid version
lets you incorporate all changes to the virtual disk onto the real disk.
Are you using any kind of this software that might be reserving disk
space while it is active? Returnil, for example, can be set to always
boot into safe mode so the only way out is to have admin rights to
change its config to not boot into safe mode and to be back to real disk
mode. SteadyState and other products do similar disk virtualizing so
all changes can be discarded but you have direct access to all other
hardware so, for example, your video game runs at full speed using the
real video card instead of an emulated one used inside virtual machines.

If you are doing regular/daily backups then you really don't need System
Restore. Disable System Restore (at least on that partition) which will
delete all restore points for that partition. Restore points won't
include you backup files, anyway, so there's no point in enabling System
Restore on that partition.
  #4  
Old October 21st 13, 01:37 AM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

"Paul" wrote in message
...
W wrote:
I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of

all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up

the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in

size.
So there are no small files on the partition that would waste empty

sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS?

What
tools might help me to explore this further?


The mystery area is probably System Volume Information.

Make sure System Restore is not "tracking" your backup drive.
WinXP tends to turn that on by default, each time a new drive
is connected. I have System Restore turned off on WinXP now.
So that's no longer an issue. Turning off System Restore,
does not remove SVI folder, neither does it disable
VSS service (which is used when backing up a partition
with a modern backup tool).

I wish there was an easy way to visualize what's in there,
so I could do more testing on this.

You could try ShadowExplorer, which is supposed to make
visible things that are being shadowed (and hidden) inside
SVI. But ShadowExplorer doesn't show you everything
stored in SVI. In WinXP, you see lots of RPxx folders
for restore points, as a major contributor to bloat.
Turning off System Restore, should make those go away
(on all your partitions, not just the one you're concerned
about right now). It really depends on whether you consider
System Restore to be worthwhile, as to whether it should
stay turned on. When you do Windows Update, that is one time
it's nice to have it turned on, just in case.

I boot a Linux LiveCD for a look inside System Volume Information.
On Vista/Win7/Win8, it's possible to trash the OS by screwing
around in that folder. So take precautions before becoming
too adventurous. I had to restore my laptop from a backup,
after one of my little experiments ran amok. And using the
Linux LiveCD was all part of that (causing the problem) :-)

One of my Linux LiveCD collection, the Knoppix 5.3.1 DVD,
mounts all partitions read-only. And while I don't use
that disc regularly any more, that's the only one I trust
not to trash stuff. It's like a gun with a safety -
you can still turn off the safety on that LiveCD environment,
and trash stuff. But it stops you, if you're a noob at it.
To go read/write, takes an extra step.


I turned off System Volume Information on the backup drive, just to be safe,
but there was nothing inside that folder.

I don't understand why you wouldn't just give Local Administrators group
READ access to System Volume Information. I have done this for a long time
on many machines and have never seen any side effect.

--
W


  #5  
Old October 21st 13, 01:41 AM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

"VanguardLH" wrote in message
...
W wrote:

I have a 1 TB NTFS partition on Windows XP that reports through Explorer
Properties dialog as having 70GB available. When I add up the size of

all
of the files on disk, there should be 350GB available.

I am very aware of cluster sizes and how many small files would take up

the
minimum cluster size, usually 4096 bytes per file. The problem is the
partition in question only holds huge backup files, minimum 1 GB in

size.
So there are no small files on the partition that would waste empty

sections
of each cluster.

I emptied the Recycle Bin, so deleted files are not accounting for this
issue.

What would account for the waste of space being reported by the OS?

What
tools might help me to explore this further?


Did you include hidden-marked files? Did you configure Windows Explorer
to show hidden-marked files?

Did you include special and OS files not shown by Windows Explorer? Did
you include the pagefile and hibernate files? You didn't explicitly
state this was a non-OS partition. Even if it is not an OS partition,
but if the partition is on a different hard disk, often users will
spread the pagefile across multiple hard disks so reads can be queued at
the same time to the pagefile along with those to the OS partition.
Maybe the pagefile portion in the problematic partition, if enabled, is
set way too huge.

Is that partition only for use by your backup program? And that backup
is which one? Some backup programs provide for snapshots. They
deliberately hide those snapshots from the file system, so what you see
in Windows Explorer won't reflect the space consumed by the snapshots.
Then there are other "restore" utilities, like Comodo's Time Machine
(don't ever try using this betaware) or DeepFreeze, that take snaphots
of the file system and hide them from the OS. If you are using
virtualized disk I/O utilities to provide a safe environment for testing
unknown or untrusted software (just the disk is virtualized and all
other hardware is real) instead of an always slower virtual machine,
like Returnil, they hide their disk space used by their replacement file
I/O handler to virtualize disk I/O. Some folks have this setup to use
half of the available free space. It isn't in use when you are not in
"safe mode" (when virtualized disk is active) so that space is
available; however, once you go into safe (virtualized disk) mode then
that space gets used. While virtualize disk I/O is active, that space
can get consumed. When you reboot, all changes to the virtual disk are
discarded and you're back to using the real disk I/O. The paid version
lets you incorporate all changes to the virtual disk onto the real disk.
Are you using any kind of this software that might be reserving disk
space while it is active? Returnil, for example, can be set to always
boot into safe mode so the only way out is to have admin rights to
change its config to not boot into safe mode and to be back to real disk
mode. SteadyState and other products do similar disk virtualizing so
all changes can be discarded but you have direct access to all other
hardware so, for example, your video game runs at full speed using the
real video card instead of an emulated one used inside virtual machines.

If you are doing regular/daily backups then you really don't need System
Restore. Disable System Restore (at least on that partition) which will
delete all restore points for that partition. Restore points won't
include you backup files, anyway, so there's no point in enabling System
Restore on that partition.


I always configure Windows Explorer to show hidden and system files. It's
one of my first setup steps. In any case, there were no large hidden or
system files. But having an application that would search the entire
partition and then sort files from largest to smallest would not be a bad
utility to have.

The drive and partition in question only holds backup files. So there is no
paging file or hibernation. It's my U partition on a separate SATA drive.
This is a "real" partition and I'm not running this OS as a virtual machine.

I did turn off System Restore.

--
W


  #6  
Old October 21st 13, 03:34 AM posted to microsoft.public.windowsxp.hardware
VanguardLH[_2_]
external usenet poster
 
Posts: 9,399
Default Disk Uses More Space Than Size of Files

"VanguardLH" wrote ...

And that backup is which one?


Still waiting on that question.

In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft

http://www.symantec.com/connect/arti...e-data-streams

While there are legitimate uses of ADS, it can also be misused.

http://www.irongeek.com/i.php?page=security/altds

That shows how using just the simple 'echo' console command that you can
add a huge file onto a text file. By redirecting stdout to the target
file but specifying a name for an ADS (the part after the colon in the
filename), all that stdout goes into the ADS. While that article looks
old, ADS is still a feature of NTFS.

You didn't say if the backup partition on the external drive is
formatted using FAT32, NTFS, exFAT, or some other file system.

You can find ADS utilities to expose the multiple streams (the blank or
no name one is the primary one or the one you normally consider the
file itself). http://www.rekenwonder.com/streamexplorer.htm is one such
utility but there are probably lots of these. I've used this one in the
past but obviously it's mostly to reveal there is an ADS on file that
you select rather than scan all your files to find which ones have one,
or more, ADS attached to them. You might want to ask in the
alt.comp.freeware newsgroup (get ready to ignore lots of noise) on what
is a good ADS explorer tool. As I recall, there was one ran from the
command line that would strip all ADS from the specified files but then
you lose any meta data they stored, like a thumbnail image. I once had
such a command-line scanner tool so I know that you'll find lots of
files that have an ADS for them but often it's trivial meta-data.

I do remember that some backup programs use the ADS to keep track of
their versioning history. Don't remember which one but recall one that
used the ADS to record if a file had already been backed up and the hash
value for the file at that time. For a subsequent incremental backup
job, it could use that meta-data to determine if it could skip an
unchanged file. The archive file attribute is not a reliable means of
determining if a file has changed or not so meta-data was used to keep
track if a file (in its current state) had already been backed up. I
even recall an anti-virus program doing that (I think it was Kaspersky)
so it could shorten its on-demand scans. If the file hadn't changed
since the last scan and it was included in the scan, info stored as
meta-data in an ADS on the file, then the AV's scan could skip that file
to eliminate wasting time rechecking a file that had already been
checked before, didn't change, so it doesn't need to be checked again.
If you used an ADS scanner to strip them from files, the AV program
would have to scan all files again.

http://www.softpedia.com/progScreens...ot-134764.html

That's an example of a scanner so you don't have to manually select
files in an explorer tool to see if they have an ADS and what is its
size. I've never used ADS Scanner. Just remember there are also
legimate uses of ADS so stripping them off means losing data which could
affect functionality within the OS or an app that handles the file.

SysInternals has their 'streams' utility to scan from a command prompt
for ADS on files. Without the -d switch, it'll list which files were
found to have an ADS. With the -d switch, it would delete the ADS file
on every matching file to the filespec you gave it. Again, you could
end up deleting more than you want, so look before deleting.
  #7  
Old October 21st 13, 06:18 AM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

"VanguardLH" wrote in message
...
"VanguardLH" wrote ...

And that backup is which one?


Still waiting on that question.


The backup files are being made by Acronis True Image.


In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft


http://www.symantec.com/connect/arti...e-data-streams

While there are legitimate uses of ADS, it can also be misused.

http://www.irongeek.com/i.php?page=security/altds

That shows how using just the simple 'echo' console command that you can
add a huge file onto a text file. By redirecting stdout to the target
file but specifying a name for an ADS (the part after the colon in the
filename), all that stdout goes into the ADS. While that article looks
old, ADS is still a feature of NTFS.


So is there a utility that will give you a view of all space being used by a
file system tree, including the ADS?


You didn't say if the backup partition on the external drive is
formatted using FAT32, NTFS, exFAT, or some other file system.


Everything is NTFS.


You can find ADS utilities to expose the multiple streams (the blank or
no name one is the primary one or the one you normally consider the
file itself). http://www.rekenwonder.com/streamexplorer.htm is one such
utility but there are probably lots of these. I've used this one in the
past but obviously it's mostly to reveal there is an ADS on file that
you select rather than scan all your files to find which ones have one,
or more, ADS attached to them. You might want to ask in the
alt.comp.freeware newsgroup (get ready to ignore lots of noise) on what
is a good ADS explorer tool. As I recall, there was one ran from the
command line that would strip all ADS from the specified files but then
you lose any meta data they stored, like a thumbnail image. I once had
such a command-line scanner tool so I know that you'll find lots of
files that have an ADS for them but often it's trivial meta-data.


Obviously looking file by file doesn't scale well to a large number of
files. You want a utility that will look across whole subtrees and provide
summaries that let you dig into specific areas for further research.


I do remember that some backup programs use the ADS to keep track of
their versioning history. Don't remember which one but recall one that
used the ADS to record if a file had already been backed up and the hash
value for the file at that time. For a subsequent incremental backup
job, it could use that meta-data to determine if it could skip an
unchanged file. The archive file attribute is not a reliable means of
determining if a file has changed or not so meta-data was used to keep
track if a file (in its current state) had already been backed up. I
even recall an anti-virus program doing that (I think it was Kaspersky)
so it could shorten its on-demand scans. If the file hadn't changed
since the last scan and it was included in the scan, info stored as
meta-data in an ADS on the file, then the AV's scan could skip that file
to eliminate wasting time rechecking a file that had already been
checked before, didn't change, so it doesn't need to be checked again.
If you used an ADS scanner to strip them from files, the AV program
would have to scan all files again.


I went ahead and installed ADS Scanner and am running it. It's an extremely
primitive tool and somewhat buggy, but at this point it is a start, thanks.

--
W


  #8  
Old October 21st 13, 06:25 AM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.


You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.

--
W


  #9  
Old October 21st 13, 08:16 AM posted to microsoft.public.windowsxp.hardware
Paul
external usenet poster
 
Posts: 18,281
Default Disk Uses More Space Than Size of Files

W wrote:
"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.


You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.


If you're dealing with NTFS, you want a copy of nfi.exe.

It too is a crude utility, but it does present all the
useful information. To be really useful, it needs post-processing
with scripts. It only works on NTFS, and there is no equivalent
for FAT32.

Still, when no other utility is available or convenient, you
use what you've got.

http://support.microsoft.com/kb/253066
http://download.microsoft.com/downlo...us/oem3sr2.zip

When you find two files that have the same set of data sectors,
those files are probably hard linked, and get double-counted
by Explorer while you're attempting to total the space. Only
the "summary pie chart" in Windows, tells you how much
space the partition is really using. Attempting to use
a hand calculator and mousing over folders in the (file) Explorer
is a waste of time. More so on Vista/Win7/Win8, as things like
hard linking aren't really used that much on something like
WinXP.

*******

There are still a few files on NTFS, that no utility
will touch. For that case, there's Linux...
Even nfi.exe doesn't list everything. Compare the
listing from Linux with Windows, for more info about
what you might be missing. It's even possible
if you use listdir in Linux and list by inode, the
fake inode will correspond to the file number
seen in nfi. But I haven't tested for that.
That's just a guess.

Paul
  #10  
Old October 21st 13, 10:33 AM posted to microsoft.public.windowsxp.hardware
VanguardLH[_2_]
external usenet poster
 
Posts: 9,399
Default Disk Uses More Space Than Size of Files

W wrote:

The backup files are being made by Acronis True Image.


Ahhh. I use that, too. You didn't happen to use their "Try & Decide"
feature, did you? T&D demands the use of the Acronis Secure Zone (ASZ),
a separate partition on a hard disk formatted as FAT32 but uses a
non-standard partition type number in the MBR.

In fact, if you try to use T&D, you get somewhat screwed in that it
demands it usurp the bootstrap code in the MBR which obviates using the
Acronis Recovery Manager that also usurps the MBR bootstrap area. Only
1 can usurp the MBR bootstrap code at a time. If you want Acronis
Recovery Manager then you cannot use T&D. If you want to use T&D, you
have to forego availability of the Acronis Recovery Manager. Of course,
using either of these obviates using any other program that wants to
usurp the MBR bootstrap area (e.g., GAG, a multi-boot manager that
resides wholly outside any partition). In older versions, Acronis
Recovery Manager and T&D were compatible. It was in the last version,
or two, they changed it so they were mutually exclusive because they
changes T&D to want to usurp the MBR bootstrap code.

If you are saving your backup images in the ASZ and also using T&D then
both the image backups and T&D's virtual disk share the same partition.
Make sure T&D is disabled (off). If it is then maybe there is a problem
with it releasing the disk space in the ASZ partition. It could be that
the space for T&D is merely "reserved". That means if T&D is disabled
that Acronis will use that space for its backups. Reserved doesn't
necessarily mean inuse.
  #11  
Old October 21st 13, 10:45 AM posted to microsoft.public.windowsxp.hardware
VanguardLH[_2_]
external usenet poster
 
Posts: 9,399
Default Disk Uses More Space Than Size of Files

W wrote:

"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.


You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.


There are utilities to add-on to Windows Explorer to let users see the
streams, if any (other than the default/primary one), attached to a
file, like:

http://www.jsware.net/jsware/sviewer.php5

Haven't used it so cannot comment on its usefulness. There are probably
other shell extensions that make it convenient to check for and view
streams on folders or files using Windows Explorer.

Gets even worse in Windows 7, and later, where Microsoft decided to use
this NTFS feature to further block file access by even admin-level users
(whether they are in the Administrators group using their own account or
even if using the Administrator account). See:

http://www.jsware.net/jsware/nt6fix.php5

Their NT6 Fix utility sounds a lot like the Take Ownership utility that
I installed that appears as a context menu entry when I right-click on a
folder or file.
  #12  
Old October 21st 13, 11:31 AM posted to microsoft.public.windowsxp.hardware
VanguardLH[_2_]
external usenet poster
 
Posts: 9,399
Default Disk Uses More Space Than Size of Files

VanguardLH wrote:

W wrote:

"VanguardLH" wrote in message
...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.


You know the implication of this is that a hacker who gains control of your
system could hide an entire encrypted partition inside the ADS of a single
file and most users would never have any clue that this existed.

Given that it would surely be useful to have a service running 24x7 that
looked for abnormally large files or ADS streams on specified partitions and
sent out a warning when any are found.


There are utilities to add-on to Windows Explorer to let users see the
streams, if any (other than the default/primary one), attached to a
file, like:

http://www.jsware.net/jsware/sviewer.php5

Haven't used it so cannot comment on its usefulness. There are probably
other shell extensions that make it convenient to check for and view
streams on folders or files using Windows Explorer.

Gets even worse in Windows 7, and later, where Microsoft decided to use
this NTFS feature to further block file access by even admin-level users
(whether they are in the Administrators group using their own account or
even if using the Administrator account). See:

http://www.jsware.net/jsware/nt6fix.php5

Their NT6 Fix utility sounds a lot like the Take Ownership utility that
I installed that appears as a context menu entry when I right-click on a
folder or file.


By the way, I just noticed the 'dir' console command has an /R switch to
indicate which folders or files have alternate streams attached to them
but that's only in Windows Vista and up. Not available back in Windows
XP.

Despite ADS being built into NTFS and available for use and misuse for
well over a decade, Microsoft is still absymal in exposing ADS to end
users by the omission of decent tools or shell extensions to Windows
Explorer included in the install of Windows. Yeah, there may be tools
you can get from Microsoft to look at streams but they are definitely
not mainstream tools documented to typical end users.

You might want to read up on StrmExt, a shell extension from Microsoft
that adds a property sheet to let you see alternate streams. I'm on
Win7 x64 so that abandoned tools is unusable to me. See:

http://www.boredomsoft.org/strmext.d...x64-windows.bs

I think the guy at the following link recompiled StrmExt for use under
64-bit versions of Windows so I might look at it:

http://www.benf.org/other/alternates...lay/index.html

I remember getting into alternate streams sometime around 2000 when I
noticed none of the anti-virus programs were interrogating the alternate
streams of files. That's when I learned about ADS and soon it dawned on
my a good place to hide a malware payload. Something would still have
to execute that payload so hopefully the AV program caught that. Yet I
didn't like the idea of quiescent malware residing on my host. I don't
remember which ones but I started to raise a stink at the top AV vendors
at that time that they must scan alternate streams on folders and files.
Even if their on-access scanner didn't scan the ADS (because it would
see the caller process as the [invoker of the] malware payload), I
wanted their on-demand scanner to spend the time to go look there. It
was like 2 years before they started to add ADS as a scan location.

http://cybercrud.net/2012/09/01/alte...-invisibility/
See the "ADS as Hidden Processes" section.

Personally I don't ever remember ever seeing something that looked like
"process1rocess2" listed in Task Manager. I'm not sure ADS was ever
intended to allow an executable payload in an alternate stream. It was
for meta *data*. Windows should never allow loading an alternate stream
into memory and then executing it yet Windows does allow just this. At
this point, Microsoft should just get rid of ADS from NTFS. No users
use it. Rare even know about it. When I mentioned it, I bet is was
something new to you. I know folks with decades of experience in the
Dev and Q&A groups and still out of 50 maybe 1 or 2 will recognize what
I'm talking about when I mention ADS. It is rarely used. Few programs
use it and they shouldn't rely on the meta data being there since moving
the file from NTFS to FAT destroys the alternate streams.
  #13  
Old October 21st 13, 12:02 PM posted to microsoft.public.windowsxp.hardware
Paul
external usenet poster
 
Posts: 18,281
Default Disk Uses More Space Than Size of Files

VanguardLH wrote:

Few programs use it and they shouldn't rely on the
meta data being there since moving
the file from NTFS to FAT destroys the alternate streams.


One version of Kaspersky, used alternate data streams for
tracking files it was monitoring. So it has been used.
In that case, a small percentage of Kaspersky users
ended up with problems somehow related to alternate streams,
and the usage of that was abandoned in the next version.

On modern Windows, alternate data streams are
supposed to be used for tracking whether you downloaded
an executable. And then you can be appropriately
warned, when attempting to execute something that
came into the machine via downloading. There are
recipes around for removing the metadata, so
Windows will stop whining about that particular file.

Paul
  #14  
Old October 21st 13, 08:48 PM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

"VanguardLH" wrote in message
...
W wrote:
The backup files are being made by Acronis True Image.


Ahhh. I use that, too. You didn't happen to use their "Try & Decide"
feature, did you? T&D demands the use of the Acronis Secure Zone (ASZ),
a separate partition on a hard disk formatted as FAT32 but uses a
non-standard partition type number in the MBR.

In fact, if you try to use T&D, you get somewhat screwed in that it
demands it usurp the bootstrap code in the MBR which obviates using the
Acronis Recovery Manager that also usurps the MBR bootstrap area. Only
1 can usurp the MBR bootstrap code at a time. If you want Acronis
Recovery Manager then you cannot use T&D. If you want to use T&D, you
have to forego availability of the Acronis Recovery Manager. Of course,
using either of these obviates using any other program that wants to
usurp the MBR bootstrap area (e.g., GAG, a multi-boot manager that
resides wholly outside any partition). In older versions, Acronis
Recovery Manager and T&D were compatible. It was in the last version,
or two, they changed it so they were mutually exclusive because they
changes T&D to want to usurp the MBR bootstrap code.

If you are saving your backup images in the ASZ and also using T&D then
both the image backups and T&D's virtual disk share the same partition.
Make sure T&D is disabled (off). If it is then maybe there is a problem
with it releasing the disk space in the ASZ partition. It could be that
the space for T&D is merely "reserved". That means if T&D is disabled
that Acronis will use that space for its backups. Reserved doesn't
necessarily mean inuse.


No, I did not use Try and Decide.

And I long ago learned to never use products that modify the MBR.

--
W


  #15  
Old October 21st 13, 09:12 PM posted to microsoft.public.windowsxp.hardware
W[_2_]
external usenet poster
 
Posts: 94
Default Disk Uses More Space Than Size of Files

"VanguardLH" wrote in message
...
"VanguardLH" wrote ...
In addition, Windows Explorer will never show you the size of Alternate
Data Streams (ADS) added to a file. For example, I can create a .txt
file whose primary data stream chews up only, say 5KB but then add an
alternate data stream that is gigabytes in size. Windows Explorer,
'dir', and other normal file utilities will only show you the size of
the primary data stream.

http://en.wikipedia.org/wiki/Alterna...ream#Microsoft


http://www.symantec.com/connect/arti...e-data-streams

I found a better utility for scanning for ADS: NirSoft
AlternateStreamView. Strangely, this utility does not agree on all of the
results with the ADS Scanner you mentioned.

Some surprising things I found:

1) Dropbox is using the ADS feature actively, and many dropbox files have up
to 4096 bytes of ADS information attached to them.

2) The AlternateStreamView shows an additional field of
"StreamAllocatedSize". In my boot partition I had a few files where the
actual ADS stream was about 1K but the ADS Allocated Size was about 65K.
Does anyone know if the "allocated size" represents actual disk space in
use?

Most of these files that AlternateStreamView reported large allocation sizes
on were not even seen by ADS Scanner.

--
W


 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off






All times are GMT +1. The time now is 01:22 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright 2004-2018 PCbanter.
The comments are property of their posters.