Drive compression happens later?

I marked an empty drive as compressed, then filled it completely full with data. I came back to it a few days later and it had a fair amount of free space. Does compression not happen on the fly?

--
You can't please everyone. But it IS possible to **** 'em ALL off at the same time.

On Mon, 07 May 2018 14:33:18 +0100, Wolf K wrote:

On 2018-05-07 06:32, Jimmy Wilkinson Knife wrote:
I marked an empty drive as compressed, then filled it completely full
with data. I came back to it a few days later and it had a fair amount
of free space. Does compression not happen on the fly?

"On the fly" often means "when the system is idle".

Do you have a link to prove this either way? I'd like to know for sure if the compression is done during writing, or if it might write uncompressed then compress later.

If it's the latter, it's not as useful as I thought for a backup disk.

--
"We hang the petty thieves and appoint the great ones to high office" - Aesop

Jimmy Wilkinson Knife wrote:
I marked an empty drive as compressed, then filled it completely full
with data. I came back to it a few days later and it had a fair amount
of free space. Does compression not happen on the fly?


It can compress in the background, if you enable the "whole drive
tick box" after the drive has files on it.

https://www.windowscentral.com/how-u...ion-windows-10

It also compresses in real time, when you copy new files to a
volume that has the tick box set.

By all means play with it, if you need a hobby.

I do *not* recommend NTFS compression for general usage.

You'll figure out why, after a while.

If you need to compress stuff, apply 7ZIP or RAR to individual
objects and manage compression yourself. 7ZIP has a "verify" function,
if you want it to verify the internal checksum on the file later.

If I had two 500GB disks, I would sooner use Dynamic Disk and
"span" the two drives, to make a 1TB scratch volume, than enable
Compression on one drive, and try to squeeze the data onto just
one drive.

Dynamic Disk is good enough for "scratch projects" where it
doesn't matter if the DD dies on the job. I would not use
Dynamic Disk for long term/archival storage.

To give an example of a "scratch project", Microsoft ICE, the
panorama stitching software, will declare part way through
the run, that it "needs a 2TB drive for temporary files
it will create during the run". To humor the software,
I could span a bunch of garbage drives, just to give the
software somewhere to write. The actual final output file from the
run, might have been 4GB to 10GB or so, so the spanned volume
can be taken apart after the run finishes.

Paul

On Mon, 07 May 2018 14:55:18 +0100, Paul wrote:

Jimmy Wilkinson Knife wrote:
I marked an empty drive as compressed, then filled it completely full
with data. I came back to it a few days later and it had a fair amount
of free space. Does compression not happen on the fly?


It can compress in the background, if you enable the "whole drive
tick box" after the drive has files on it.

https://www.windowscentral.com/how-u...ion-windows-10

It also compresses in real time, when you copy new files to a
volume that has the tick box set.

By all means play with it, if you need a hobby.

I do *not* recommend NTFS compression for general usage.

You'll figure out why, after a while.

If you need to compress stuff, apply 7ZIP or RAR to individual
objects and manage compression yourself. 7ZIP has a "verify" function,
if you want it to verify the internal checksum on the file later.

If I had two 500GB disks, I would sooner use Dynamic Disk and
"span" the two drives, to make a 1TB scratch volume, than enable
Compression on one drive, and try to squeeze the data onto just
one drive.

Dynamic Disk is good enough for "scratch projects" where it
doesn't matter if the DD dies on the job. I would not use
Dynamic Disk for long term/archival storage.

To give an example of a "scratch project", Microsoft ICE, the
panorama stitching software, will declare part way through
the run, that it "needs a 2TB drive for temporary files
it will create during the run". To humor the software,
I could span a bunch of garbage drives, just to give the
software somewhere to write. The actual final output file from the
run, might have been 4GB to 10GB or so, so the spanned volume
can be taken apart after the run finishes.

I'm using drive compression as a stop-gap until I get another drive to double the space. I marked the drive as compressed BEFORE I copied ANY data to it. Would it still compress after the files were written? Because I filled the drive to the brim, then 2 days later discovered 5-10% free.

--
Keyboards used to be expensive and beer used to be cheap.
Now beer is expensive and keyboards are cheap.
Conclusion, it's still bad to spill beer on your keyboard.

On Mon, 07 May 2018 15:02:42 +0100, Wolf K wrote:

On 2018-05-07 09:42, Jimmy Wilkinson Knife wrote:
On Mon, 07 May 2018 14:33:18 +0100, Wolf K wrote:

On 2018-05-07 06:32, Jimmy Wilkinson Knife wrote:
I marked an empty drive as compressed, then filled it completely full
with data. I came back to it a few days later and it had a fair amount
of free space. Does compression not happen on the fly?

"On the fly" often means "when the system is idle".

Do you have a link to prove this either way? I'd like to know for sure
if the compression is done during writing, or if it might write
uncompressed then compress later.

No link, it was just a suggestion for further look-see. If you filled up
the disk "completely" in one go, then immediately switched to another
task, the compression may not have finished. Or it may not have released
any processing space it needed on the compressed disk. Or maybe your
backup plan allows deletion of aged files that no longer exist on the
source disk. I'm sure someone with more expertise than I will be able to
sort this out further.

I simply coped the full capacity of the disk in one go. Nothing would have deleted any files.

If it's the latter, it's not as useful as I thought for a backup disk.

AFAIK, uncompression is done as the disk is read. AFAIK, "on the fly"

That's a phrase I used, I'm not quoting it from anything. I just assumed it would be done just before writing the data to the disk.

usually means "background process", ie, the OS has to schedule this
processes with others so that they all get more or less equal time
overall, depending on their relative importance.

Anyhow, I don't think you should give up on a compressed disk just
because you saw a bit of empty space. Good luck with further investigations.

I'm only using it as it's slightly too full. I'm getting more real disk, as it's bloody slow with older processors.

--
Never take life seriously. Nobody gets out alive anyway

Jimmy Wilkinson Knife wrote:
On Mon, 07 May 2018 14:55:18 +0100, Paul wrote:

Jimmy Wilkinson Knife wrote:
I marked an empty drive as compressed, then filled it completely full
with data. I came back to it a few days later and it had a fair amount
of free space. Does compression not happen on the fly?


It can compress in the background, if you enable the "whole drive
tick box" after the drive has files on it.

https://www.windowscentral.com/how-u...ion-windows-10

It also compresses in real time, when you copy new files to a
volume that has the tick box set.

By all means play with it, if you need a hobby.

I do *not* recommend NTFS compression for general usage.

You'll figure out why, after a while.

If you need to compress stuff, apply 7ZIP or RAR to individual
objects and manage compression yourself. 7ZIP has a "verify" function,
if you want it to verify the internal checksum on the file later.

If I had two 500GB disks, I would sooner use Dynamic Disk and
"span" the two drives, to make a 1TB scratch volume, than enable
Compression on one drive, and try to squeeze the data onto just
one drive.

Dynamic Disk is good enough for "scratch projects" where it
doesn't matter if the DD dies on the job. I would not use
Dynamic Disk for long term/archival storage.

To give an example of a "scratch project", Microsoft ICE, the
panorama stitching software, will declare part way through
the run, that it "needs a 2TB drive for temporary files
it will create during the run". To humor the software,
I could span a bunch of garbage drives, just to give the
software somewhere to write. The actual final output file from the
run, might have been 4GB to 10GB or so, so the spanned volume
can be taken apart after the run finishes.

I'm using drive compression as a stop-gap until I get another drive to
double the space. I marked the drive as compressed BEFORE I copied ANY
data to it. Would it still compress after the files were written?
Because I filled the drive to the brim, then 2 days later discovered
5-10% free.


When Explorer "considers" a file transfer, it won't proceed unless
there is space. You yourself may know that your 2TB of files
will compress nicely to fit the 1TB drive. But Explorer doesn't
know that. It won't let you copy more than 1TB of files to the
1TB compressing drive. Then, when the actual files only took
0.55TB to store, the next transfer you attempt will have room
to allow a 0.45TB transfer. In other words, you will
asymptotically be filling up the drive. It may take
many Explorer-mediated file transfers, to fill it to
the absolute brim.

There is no option such as "hey, Explorer, take your
best shot with these 2TB of files and just die if
you run out of space". They didn't set it up that way.
It uses "uncompressed, worst case behavior" for
estimating available space.

*******

On the other hand, you could ask Robocopy to do it,
and it might not care :-)

(Note: mirror mode is dangerous, and effectively erases F:
when copying. If you accidentally type the wrong drive letter
for the destination, you will be very sorry. Robocopy is
a folder-to-folder copy agent, but will accept a drive
as a folder specification.)

robocopy I:\ F:\ /mir /copy:datso /dcopy:t /r:3 /w:2 /zb /np /tee /v /log:IF_out.log

Robocopy keeps a log entry for each file handled. In
mirror mode, it'll say "skip" if the identical file is
already on the drive in the specified folder. It might
know this, via date stamp and byte size. It doesn't
really know whether the files are exactly alike, before
skipping, and only uses the "normal amount of checking"
via size and date stamp.

If copying C: say (something with ~62 Junction points), you
may want to use the command line option to skip junctions,
and you'll want to include that in your command parameters.
Not that I recommend copying C: this way - as the disk
created probably wouldn't be boot-able. I only used C: as
an example of a volume with known Junction Points onboard,
for which you would want to include that option in the
command line parameters. If a disk had Junction Point
entries, and you didn't use /Xj or whatever, then the
command will bomb when it hits one.

robocopy /?

HTH,
Paul

On Mon, 07 May 2018 15:28:30 +0100, Paul wrote:

Jimmy Wilkinson Knife wrote:
On Mon, 07 May 2018 14:55:18 +0100, Paul wrote:

Jimmy Wilkinson Knife wrote:
I marked an empty drive as compressed, then filled it completely full
with data. I came back to it a few days later and it had a fair amount
of free space. Does compression not happen on the fly?


It can compress in the background, if you enable the "whole drive
tick box" after the drive has files on it.

https://www.windowscentral.com/how-u...ion-windows-10

It also compresses in real time, when you copy new files to a
volume that has the tick box set.

By all means play with it, if you need a hobby.

I do *not* recommend NTFS compression for general usage.

You'll figure out why, after a while.

If you need to compress stuff, apply 7ZIP or RAR to individual
objects and manage compression yourself. 7ZIP has a "verify" function,
if you want it to verify the internal checksum on the file later.

If I had two 500GB disks, I would sooner use Dynamic Disk and
"span" the two drives, to make a 1TB scratch volume, than enable
Compression on one drive, and try to squeeze the data onto just
one drive.

Dynamic Disk is good enough for "scratch projects" where it
doesn't matter if the DD dies on the job. I would not use
Dynamic Disk for long term/archival storage.

To give an example of a "scratch project", Microsoft ICE, the
panorama stitching software, will declare part way through
the run, that it "needs a 2TB drive for temporary files
it will create during the run". To humor the software,
I could span a bunch of garbage drives, just to give the
software somewhere to write. The actual final output file from the
run, might have been 4GB to 10GB or so, so the spanned volume
can be taken apart after the run finishes.

I'm using drive compression as a stop-gap until I get another drive to
double the space. I marked the drive as compressed BEFORE I copied ANY
data to it. Would it still compress after the files were written?
Because I filled the drive to the brim, then 2 days later discovered
5-10% free.


When Explorer "considers" a file transfer, it won't proceed unless
there is space. You yourself may know that your 2TB of files
will compress nicely to fit the 1TB drive. But Explorer doesn't
know that. It won't let you copy more than 1TB of files to the
1TB compressing drive. Then, when the actual files only took
0.55TB to store, the next transfer you attempt will have room
to allow a 0.45TB transfer. In other words, you will
asymptotically be filling up the drive. It may take
many Explorer-mediated file transfers, to fill it to
the absolute brim.

There is no option such as "hey, Explorer, take your
best shot with these 2TB of files and just die if
you run out of space". They didn't set it up that way.
It uses "uncompressed, worst case behavior" for
estimating available space.

No, the files were copied one by one by xxcopy (that's 2 x's). It won't stop unless the single file it's currently copying doesn't fit. The drive was completely filled, then after a couple of days got more space on it, which suggests some compression must have taken place after the writing.


--
CONGRESS.SYS corrupted... Re-boot Washington D.C. (Y/N)?

In article
"Jimmy Wilkinson Knife" wrote:

I marked an empty drive as compressed, then filled it completely full with data. I came back to it a few days later and it had a fair amount of free space. Does compression not happen on the fly?


It happens as first written to disk. Unless you use cheap 3rd party
tool.

On Sun, 13 May 2018 21:57:22 +0200 (CEST), "Anonymous Remailer
(austria)" wrote:


In article
"Jimmy Wilkinson Knife" wrote:

I marked an empty drive as compressed, then filled it completely full with data. I came back to it a few days later and it had a fair amount of free space. Does compression not happen on the fly?


It happens as first written to disk. Unless you use cheap 3rd party
tool.

The argument is that the CPU can compress the data faster than the
disk can write it so compressing on the fly saves time. Same for
decompressing.
--

Regards,

Eric Stevens

Eric Stevens wrote:
On Sun, 13 May 2018 21:57:22 +0200 (CEST), "Anonymous Remailer
(austria)" wrote:

In article
"Jimmy Wilkinson Knife" wrote:
I marked an empty drive as compressed, then filled it completely full with data. I came back to it a few days later and it had a fair amount of free space. Does compression not happen on the fly?

It happens as first written to disk. Unless you use cheap 3rd party
tool.

The argument is that the CPU can compress the data faster than the
disk can write it so compressing on the fly saves time. Same for
decompressing.

It does compress on the fly (for sure), when you
click the "compress" tick box in the drive letter
properties dialog, and it asks you whether you
want to compress the entire hierarchy on the disk.
It then proceeds to traverse the tree and compress.

I don't know if it does that on a copy (i.e. lazy compression).
That would make a mess of the disk. Especially if the
"compression writer" was competing with the "file copy writer".
It would be fragment-city.

Paul