Interesting discrepencies

First off: Win 10 1809 fully updated

I found some good deals on UDB thumb drives, so I thought I would do some
archival backups of some things that are not vital, but are essentially
irreplaceable.

I have one directory where if I right-click and do 'Properties' it says the
direcctory is 21+ GB. But Windirstat shows 521 GB of data in that
directory.

Can anyone explain what is going on here?

lonelydad wrote:

I have one directory where if I right-click and do 'Properties' it says the
direcctory is 21+ GB. But Windirstat shows 521 GB of data in that
directory.

Don't tell us the name of the folder ...
Can anyone explain what is going on here?

If it's %windir%\winsxs then the answer is hard links

In article ,
says...

First off: Win 10 1809 fully updated

I found some good deals on UDB thumb drives, so I thought I would do some
archival backups of some things that are not vital, but are essentially
irreplaceable.

I have one directory where if I right-click and do 'Properties' it says the
direcctory is 21+ GB. But Windirstat shows 521 GB of data in that
directory.

Can anyone explain what is going on here?

USB thumb drives are one of the less reliable media for long-term
storage. Storage is pretty cheap, so for "belt & braces", consider a
cheap spinning disk, a cheap SSD, and online storage - copy everything
to all three!

--

Phil, London

On 15/08/2019 01.26, lonelydad wrote:
First off: Win 10 1809 fully updated

I found some good deals on UDB thumb drives, so I thought I would do some
archival backups of some things that are not vital, but are essentially
irreplaceable.

Bad idea, then. Do not store things that are important on an USB stick,
use real hard disks instead. Rotating rust or SSD. Even Blue Ray DVDs of
archival quality.

--
Cheers, Carlos.

Andy Burns wrote in
:

lonelydad wrote:

I have one directory where if I right-click and do 'Properties' it
says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in
that directory.

Don't tell us the name of the folder ...

It's one I created with no links to any Windows permanent folder. 'My
Tidbits'

lonelydad wrote:
Andy Burns wrote in
:

lonelydad wrote:

I have one directory where if I right-click and do 'Properties' it
says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in
that directory.
Don't tell us the name of the folder ...

It's one I created with no links to any Windows permanent folder. 'My
Tidbits'


If it really is 521GB, you might as well back up the whole C:
drive, for thoroughness. A Macrium image of the disk, will
ensure you don't miss anything.

And a USB stick ? Yikes. To hold 521GB, that would need to be
a 1TB stick. Just make sure you paid more than $10 for it, as
the $10 ones are counterfeit.

A 1TB hard drive, is dirt cheap (it's a one platter drive).

A SATA SSD of that size, still costs money, but
at least it has a slight edge on longevity compared
to some of the USB sticks out there.

USB sticks are for sneaknet, for transporting files
to remote locations. They're not suitable for archival
storage. Back in the SLC or MLC era, maybe. But
the track record here (two TLC USB flash failed in
around a year of light usage), I wouldn't touch
this idea with a barge pole. And when the QLC
drains into the retail USB stick channel, it'll really
be hell then.

Paul

In article , lid
says...

On 15/08/2019 01.26, lonelydad wrote:
First off: Win 10 1809 fully updated

I found some good deals on UDB thumb drives, so I thought I would do some
archival backups of some things that are not vital, but are essentially
irreplaceable.

Bad idea, then. Do not store things that are important on an USB stick,
use real hard disks instead. Rotating rust or SSD. Even Blue Ray DVDs of
archival quality.

I burn everything I'd ever want to keep long term to dvd. I can still
read dvds I created well over 19 years ago now (unlike collection of
5.25 $ 3.5" floppies won't even format now). There's also a second copy
on removable hard disk and for the important stuff even a third separate
exeternal hard disk copy. There's also image backups but there's usually
little data contained in them.

I got burned once back in Win3 days so try to insure never again.

Paul wrote in :

lonelydad wrote:
Andy Burns wrote in
:

lonelydad wrote:

I have one directory where if I right-click and do 'Properties' it
says the direcctory is 21+ GB. But Windirstat shows 521 GB of data
in
that directory.
Don't tell us the name of the folder ...

It's one I created with no links to any Windows permanent folder. 'My
Tidbits'


If it really is 521GB, you might as well back up the whole C:
drive, for thoroughness. A Macrium image of the disk, will
ensure you don't miss anything.


I don't keep any data on my C:\ drive. The directory in question is on a
2TB hard drive [D:\]

Given the actual size of the directory in question, I have no intention
of storing it on one of my 512GB USB sticks. I'll use the WD 1TB My
Passport drive I have and copy it there.

Right now the burning issue is why the discrepency? So far, no one has
addressed the big issue, that File Explorer is reporting a 521GB
directory as only 21GB.

lonelydad wrote:

Right now the burning issue is why the discrepency?

You could look for hard links, reparse points and other NTFS oddities, I
believe sysinternals has some tools for that ...

lonelydad wrote:
Paul wrote in :

lonelydad wrote:
Andy Burns wrote in
:

lonelydad wrote:

I have one directory where if I right-click and do 'Properties' it
says the direcctory is 21+ GB. But Windirstat shows 521 GB of data
in
that directory.
Don't tell us the name of the folder ...
It's one I created with no links to any Windows permanent folder. 'My
Tidbits'

If it really is 521GB, you might as well back up the whole C:
drive, for thoroughness. A Macrium image of the disk, will
ensure you don't miss anything.


I don't keep any data on my C:\ drive. The directory in question is on a
2TB hard drive [D:\]

Given the actual size of the directory in question, I have no intention
of storing it on one of my 512GB USB sticks. I'll use the WD 1TB My
Passport drive I have and copy it there.

Right now the burning issue is why the discrepency? So far, no one has
addressed the big issue, that File Explorer is reporting a 521GB
directory as only 21GB.

The summary is, whether it's a hardlink problem (double-counting
problem), or a Junction Point problem (File Explorer won't
descend a Junction Point), you cannot be assured that doing
"Properties" on a folder will give a correct answer. You
would have received a correct answer in the year 2003, but in
2019, not so much.

Not even WinDirStat is assured of getting a correct answer.

*No* utility lists all the filenum entries in NTFS.

Everything.exe would come close. But is probably still
four items short. It's not a good idea for example,
to be poking around System Volume Information.

*******

The one indicator which is accurate, is when you do
"Properties" on C: at the top level, and get that
"circle thingy" which says how full it is, that
actually works. And the reason that works, is that
is a "simple cluster counter". It knows how many
clusters are currently in use.

As an example, WinSXS might contain say, 6GB to 10GB of files.
A naive user would say to themselves "if I deleted WinSXS,
I will save 6 to 10GB, just like that". Bzzzt. Wrong.
If you do Properties on C: before and after, the savings
is only 500MB. All you're doing, is deleting thousands
of hardlink filename entries, rather than deleting
the files themselves.

However, if you "copy" WinSXS (don't do that), sure,
the copy takes real space, 6GB to 10GB of space. It would
also cause a copy which is "unsuited for purpose", as you
can't put it back in place of the deleted WinSXS, because
all the links would be broken. WinSXS is part of component
based servicing (CBS), and its primary function is the
hardlinks themselves. It's not intended for "Wally to be
browsing the folder", that's not why it is there. It's
to allow some Microsoft software to "maintain the OS
in a known state".

On Win98, you would expect most of the files and
facilities to be "at face value", easily manipulated
by the user, logical in nature and making sense.
But that's not how Win10 works.

Win10 is a rats nest of hard-to-understand hacks.

And it isn't particularly a good design.

The design of File Explorer, the nature of the
fields put in the Properties window for Folders
and Files, has not kept up with the evolution
of the (abuse) of the file system. And even if
there was a separate "shared" field in the Properties
box, people would not know what to do with that
information. It's not clear how best to help
the end-user understand the storage situation.

Maybe someone else knows the answer to this, but
the file copying is pretty naive. If a file is
hardlinked, the destination file will be an ordinary
file. If the source file was "sparse", the destination
file would likely be "ordinary" and "quite wasteful".

Copying the following file is especially nasty. As when
you copy it, it takes gigabytes of space, whereas
the storage at the moment actually requires, is
only 4MB. There is a utility you can use, to convert
it back to sparse format, but by then the damage
is done (takes a long time to copy, a long time
to convert back to sparse after the copy completes).

https://i.postimg.cc/TwyTgfzm/file.gif

I wonder what WinDirStat would say about that one.

If I ask File Explorer to copy that to a USB key,
4MB is all the space it really needs, yet if I
copy that file to a 32GB stick, it'll say "not enough
space to complete this command". Because the file
needs 57GB or so as far as File Explorer is concerned.

Linux has:

cp --sparse=always somefile.bin someotherlocation.bin

and that one would only need 4MB for the example file above.
I don't think Windows "copy" has that refinement. But
after the fact, there is a utility to "squeeze out the
excess" and make it take 4MB again.

Paul

Andy Burns wrote:
lonelydad wrote:

Right now the burning issue is why the discrepency?

You could look for hard links, reparse points and other NTFS oddities, I
believe sysinternals has some tools for that ...


I think you would have a hard time getting
the size issue right on the first try (as a re-write).

A guess at a first cut at a File Explorer properties, would be

Size 57,000,000,000 Naive size counting everything
Shared 9,000,000,000 Items in tree which are hardlinked.
This amount should be "treated with suspicion" if
adding the "size" values from several
folders in a spreadsheet.
DoubleCount 1,000,000 This would be two files which are
hardlinked to each other, but are in the same tree.
Subtract this from Size for Sizing reasons
(as part of working out Size On Disk).
But for a file copy, "Size" is
the amount of space needed.

Because Size information is used for more than one
reason, a single number and a naive interpretation
just won't do. It's not enough.

The File Copy will always get the right answer,
as it ensures that every estimate is conservative,
and it "expands" files that might have had
compact representations otherwise. And what that
approach does for you, is increase the number
of occasions where the copy fails to start
because "insufficient space is available".

When you're looking at your C: volume, you start
with the pie chart. If the pie chart says the disk
has 521GB+ of files, and you know it's "just an OS install"
that takes 20GB, you know there are about 500GB of
files on there *somewhere*, unaccounted for. And that's
when you'd be reaching for windirstat or the like.
Or maybe even, using the "everything.exe" file list dump.

Everything.exe -create-filelist C_filelist.txt "C:"

In my notes it says (as a breadcrumb for me)

#Everything.exe missed the files in lxss\temp

which means that while Everything.exe reads the $MFT
in about two seconds, and at that instant in time
it *can* see lxss\temp, during the next 15 seconds
it tries to stat() the sizes of all files. It walks the tree.
It fails to have permission to enter lxss\temp (even as administrator),
and instead of leaving the fields blank, it just removes
any mention of lxss\temp. (lxss\temp is likely part of the
Ubuntu install from the Windows 10 Store, so you might
not have it anyway, on your disk.)

Everything.exe should then also have trouble
telling you what is inside System Volume Information.
Even Macrium is unlikely to back up everything in SVI.
Macrium does not back up Windows.edb (search indexer database).

Really, your C: partition is some flavor of "Swiss Cheese"
in terms of good accounting principles. There are holes
all over the place. Lots of exceptions to memorize.
Don't walk into any left-over named pipes in the Crypto
folder - that's like walking in cow patties. How can an
OS forget to close the named pipes :-/ ?

Even if you're writing your own utility, you'll have
a lot of fun with this stuff. Never knowing for sure,
how close your size estimate is to the truth.

Or for that matter, never knowing how complete your
file copy is.

And *this* is why we use Macrium with Windows 10.
Never having to say you were sorry...
A cluster level backup, is bound to get the goods.
If you have items you want to make sure you got,
do it that way.

Paul

In article ,
says...

Paul wrote in :

lonelydad wrote:
Andy Burns wrote in
:

lonelydad wrote:

I have one directory where if I right-click and do 'Properties' it
says the direcctory is 21+ GB. But Windirstat shows 521 GB of data
in
that directory.
Don't tell us the name of the folder ...

It's one I created with no links to any Windows permanent folder. 'My
Tidbits'


If it really is 521GB, you might as well back up the whole C:
drive, for thoroughness. A Macrium image of the disk, will
ensure you don't miss anything.


I don't keep any data on my C:\ drive. The directory in question is on a
2TB hard drive [D:\]

Given the actual size of the directory in question, I have no intention
of storing it on one of my 512GB USB sticks. I'll use the WD 1TB My
Passport drive I have and copy it there.

Right now the burning issue is why the discrepency? So far, no one has
addressed the big issue, that File Explorer is reporting a 521GB
directory as only 21GB.

Are you sure you are viewing ALL the files? Windows can and does hide
some by default.

Carlos wrote:
Do not store things that are important on an USB stick,

Short-Term storage is what I care about,
and that's where USB sticks Excel.

Just today, and many times recently,
USB backups ( RoboCopy mirroring ) have saved my neck.

When my motherboard died,
I took my USB stick to a printshop,
so I could print vital information.

Without it, I would've been in deep, deep doo-doo.

I'm getting 3.5 GigaBytes/Sec Read|Write on my TeraByte NVMe drive.
..4/.31 Read/Write GigaBytes/Sec on my half TeraByte USB 3.1 stick.
CrystalDiskMark, default settings.

My new SSD and USB stick are both clones of my older SSD;
so I can boot off any one of them, should things go wrong,
knowing that it'll work ( like a "restore point" ).

Front panel USBs: 2.0, 2.0, 3.0
Back: 2.0, 2.0, 3.1, 3.1, 3.1 Gen2, 3.1 Gen2 TypeC.