If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Rate Thread | Display Modes |
#1
|
|||
|
|||
Interesting discrepencies
First off: Win 10 1809 fully updated
I found some good deals on UDB thumb drives, so I thought I would do some archival backups of some things that are not vital, but are essentially irreplaceable. I have one directory where if I right-click and do 'Properties' it says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in that directory. Can anyone explain what is going on here? |
Ads |
#2
|
|||
|
|||
Interesting discrepencies
lonelydad wrote:
I have one directory where if I right-click and do 'Properties' it says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in that directory. Don't tell us the name of the folder ... Can anyone explain what is going on here? If it's %windir%\winsxs then the answer is hard links |
#3
|
|||
|
|||
Interesting discrepencies
|
#4
|
|||
|
|||
Interesting discrepencies
On 15/08/2019 01.26, lonelydad wrote:
First off: Win 10 1809 fully updated I found some good deals on UDB thumb drives, so I thought I would do some archival backups of some things that are not vital, but are essentially irreplaceable. Bad idea, then. Do not store things that are important on an USB stick, use real hard disks instead. Rotating rust or SSD. Even Blue Ray DVDs of archival quality. -- Cheers, Carlos. |
#5
|
|||
|
|||
Interesting discrepencies
Andy Burns wrote in
: lonelydad wrote: I have one directory where if I right-click and do 'Properties' it says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in that directory. Don't tell us the name of the folder ... It's one I created with no links to any Windows permanent folder. 'My Tidbits' |
#6
|
|||
|
|||
Interesting discrepencies
lonelydad wrote:
Andy Burns wrote in : lonelydad wrote: I have one directory where if I right-click and do 'Properties' it says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in that directory. Don't tell us the name of the folder ... It's one I created with no links to any Windows permanent folder. 'My Tidbits' If it really is 521GB, you might as well back up the whole C: drive, for thoroughness. A Macrium image of the disk, will ensure you don't miss anything. And a USB stick ? Yikes. To hold 521GB, that would need to be a 1TB stick. Just make sure you paid more than $10 for it, as the $10 ones are counterfeit. A 1TB hard drive, is dirt cheap (it's a one platter drive). A SATA SSD of that size, still costs money, but at least it has a slight edge on longevity compared to some of the USB sticks out there. USB sticks are for sneaknet, for transporting files to remote locations. They're not suitable for archival storage. Back in the SLC or MLC era, maybe. But the track record here (two TLC USB flash failed in around a year of light usage), I wouldn't touch this idea with a barge pole. And when the QLC drains into the retail USB stick channel, it'll really be hell then. Paul |
#7
|
|||
|
|||
Interesting discrepencies
|
#8
|
|||
|
|||
Interesting discrepencies
Paul wrote in :
lonelydad wrote: Andy Burns wrote in : lonelydad wrote: I have one directory where if I right-click and do 'Properties' it says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in that directory. Don't tell us the name of the folder ... It's one I created with no links to any Windows permanent folder. 'My Tidbits' If it really is 521GB, you might as well back up the whole C: drive, for thoroughness. A Macrium image of the disk, will ensure you don't miss anything. I don't keep any data on my C:\ drive. The directory in question is on a 2TB hard drive [D:\] Given the actual size of the directory in question, I have no intention of storing it on one of my 512GB USB sticks. I'll use the WD 1TB My Passport drive I have and copy it there. Right now the burning issue is why the discrepency? So far, no one has addressed the big issue, that File Explorer is reporting a 521GB directory as only 21GB. |
#9
|
|||
|
|||
Interesting discrepencies
lonelydad wrote:
Right now the burning issue is why the discrepency? You could look for hard links, reparse points and other NTFS oddities, I believe sysinternals has some tools for that ... |
#10
|
|||
|
|||
Interesting discrepencies
lonelydad wrote:
Paul wrote in : lonelydad wrote: Andy Burns wrote in : lonelydad wrote: I have one directory where if I right-click and do 'Properties' it says the direcctory is 21+ GB. But Windirstat shows 521 GB of data in that directory. Don't tell us the name of the folder ... It's one I created with no links to any Windows permanent folder. 'My Tidbits' If it really is 521GB, you might as well back up the whole C: drive, for thoroughness. A Macrium image of the disk, will ensure you don't miss anything. I don't keep any data on my C:\ drive. The directory in question is on a 2TB hard drive [D:\] Given the actual size of the directory in question, I have no intention of storing it on one of my 512GB USB sticks. I'll use the WD 1TB My Passport drive I have and copy it there. Right now the burning issue is why the discrepency? So far, no one has addressed the big issue, that File Explorer is reporting a 521GB directory as only 21GB. The summary is, whether it's a hardlink problem (double-counting problem), or a Junction Point problem (File Explorer won't descend a Junction Point), you cannot be assured that doing "Properties" on a folder will give a correct answer. You would have received a correct answer in the year 2003, but in 2019, not so much. Not even WinDirStat is assured of getting a correct answer. *No* utility lists all the filenum entries in NTFS. Everything.exe would come close. But is probably still four items short. It's not a good idea for example, to be poking around System Volume Information. ******* The one indicator which is accurate, is when you do "Properties" on C: at the top level, and get that "circle thingy" which says how full it is, that actually works. And the reason that works, is that is a "simple cluster counter". It knows how many clusters are currently in use. As an example, WinSXS might contain say, 6GB to 10GB of files. A naive user would say to themselves "if I deleted WinSXS, I will save 6 to 10GB, just like that". Bzzzt. Wrong. If you do Properties on C: before and after, the savings is only 500MB. All you're doing, is deleting thousands of hardlink filename entries, rather than deleting the files themselves. However, if you "copy" WinSXS (don't do that), sure, the copy takes real space, 6GB to 10GB of space. It would also cause a copy which is "unsuited for purpose", as you can't put it back in place of the deleted WinSXS, because all the links would be broken. WinSXS is part of component based servicing (CBS), and its primary function is the hardlinks themselves. It's not intended for "Wally to be browsing the folder", that's not why it is there. It's to allow some Microsoft software to "maintain the OS in a known state". On Win98, you would expect most of the files and facilities to be "at face value", easily manipulated by the user, logical in nature and making sense. But that's not how Win10 works. Win10 is a rats nest of hard-to-understand hacks. And it isn't particularly a good design. The design of File Explorer, the nature of the fields put in the Properties window for Folders and Files, has not kept up with the evolution of the (abuse) of the file system. And even if there was a separate "shared" field in the Properties box, people would not know what to do with that information. It's not clear how best to help the end-user understand the storage situation. Maybe someone else knows the answer to this, but the file copying is pretty naive. If a file is hardlinked, the destination file will be an ordinary file. If the source file was "sparse", the destination file would likely be "ordinary" and "quite wasteful". Copying the following file is especially nasty. As when you copy it, it takes gigabytes of space, whereas the storage at the moment actually requires, is only 4MB. There is a utility you can use, to convert it back to sparse format, but by then the damage is done (takes a long time to copy, a long time to convert back to sparse after the copy completes). https://i.postimg.cc/TwyTgfzm/file.gif I wonder what WinDirStat would say about that one. If I ask File Explorer to copy that to a USB key, 4MB is all the space it really needs, yet if I copy that file to a 32GB stick, it'll say "not enough space to complete this command". Because the file needs 57GB or so as far as File Explorer is concerned. Linux has: cp --sparse=always somefile.bin someotherlocation.bin and that one would only need 4MB for the example file above. I don't think Windows "copy" has that refinement. But after the fact, there is a utility to "squeeze out the excess" and make it take 4MB again. Paul |
#11
|
|||
|
|||
Interesting discrepencies
Andy Burns wrote:
lonelydad wrote: Right now the burning issue is why the discrepency? You could look for hard links, reparse points and other NTFS oddities, I believe sysinternals has some tools for that ... I think you would have a hard time getting the size issue right on the first try (as a re-write). A guess at a first cut at a File Explorer properties, would be Size 57,000,000,000 Naive size counting everything Shared 9,000,000,000 Items in tree which are hardlinked. This amount should be "treated with suspicion" if adding the "size" values from several folders in a spreadsheet. DoubleCount 1,000,000 This would be two files which are hardlinked to each other, but are in the same tree. Subtract this from Size for Sizing reasons (as part of working out Size On Disk). But for a file copy, "Size" is the amount of space needed. Because Size information is used for more than one reason, a single number and a naive interpretation just won't do. It's not enough. The File Copy will always get the right answer, as it ensures that every estimate is conservative, and it "expands" files that might have had compact representations otherwise. And what that approach does for you, is increase the number of occasions where the copy fails to start because "insufficient space is available". When you're looking at your C: volume, you start with the pie chart. If the pie chart says the disk has 521GB+ of files, and you know it's "just an OS install" that takes 20GB, you know there are about 500GB of files on there *somewhere*, unaccounted for. And that's when you'd be reaching for windirstat or the like. Or maybe even, using the "everything.exe" file list dump. Everything.exe -create-filelist C_filelist.txt "C:" In my notes it says (as a breadcrumb for me) #Everything.exe missed the files in lxss\temp which means that while Everything.exe reads the $MFT in about two seconds, and at that instant in time it *can* see lxss\temp, during the next 15 seconds it tries to stat() the sizes of all files. It walks the tree. It fails to have permission to enter lxss\temp (even as administrator), and instead of leaving the fields blank, it just removes any mention of lxss\temp. (lxss\temp is likely part of the Ubuntu install from the Windows 10 Store, so you might not have it anyway, on your disk.) Everything.exe should then also have trouble telling you what is inside System Volume Information. Even Macrium is unlikely to back up everything in SVI. Macrium does not back up Windows.edb (search indexer database). Really, your C: partition is some flavor of "Swiss Cheese" in terms of good accounting principles. There are holes all over the place. Lots of exceptions to memorize. Don't walk into any left-over named pipes in the Crypto folder - that's like walking in cow patties. How can an OS forget to close the named pipes :-/ ? Even if you're writing your own utility, you'll have a lot of fun with this stuff. Never knowing for sure, how close your size estimate is to the truth. Or for that matter, never knowing how complete your file copy is. And *this* is why we use Macrium with Windows 10. Never having to say you were sorry... A cluster level backup, is bound to get the goods. If you have items you want to make sure you got, do it that way. Paul |
#12
|
|||
|
|||
Interesting discrepencies
|
#13
|
|||
|
|||
I took my USB stick to a printshop.
Carlos wrote:
Do not store things that are important on an USB stick, Short-Term storage is what I care about, and that's where USB sticks Excel. Just today, and many times recently, USB backups ( RoboCopy mirroring ) have saved my neck. When my motherboard died, I took my USB stick to a printshop, so I could print vital information. Without it, I would've been in deep, deep doo-doo. I'm getting 3.5 GigaBytes/Sec Read|Write on my TeraByte NVMe drive. ..4/.31 Read/Write GigaBytes/Sec on my half TeraByte USB 3.1 stick. CrystalDiskMark, default settings. My new SSD and USB stick are both clones of my older SSD; so I can boot off any one of them, should things go wrong, knowing that it'll work ( like a "restore point" ). Front panel USBs: 2.0, 2.0, 3.0 Back: 2.0, 2.0, 3.1, 3.1, 3.1 Gen2, 3.1 Gen2 TypeC. |
Thread Tools | |
Display Modes | Rate This Thread |
|
|