RAR Files

On Mon, 24 Jun 2019 17:47:24 -0400, Paul wrote:

To add reliability to any packaging, you can use Parchive
techniques from the outside. I don't know if anyone has
put this into, say, a multi-part archive as a means to
improving reliability.

https://en.wikipedia.org/wiki/Parchive

Well, like you said, par files are computed after the fact, when the
archive has already been created.

That was also proposed as a mechanism for enhancing
data stored on CDs or DVDs, but the problem with that
concept, is when the "error rate" gets high enough
on optical media, the laser refuses to track the spiral,
and you can't pull anything off the disc. So Parchive
for optical media is a waste of time, just because of
error characteristic (tolerates errors well, until
the laser loses lock).

I wouldn't call it a waste of time. You don't have to wait until the
read error rate gets "high enough", but if you do want to plan for that
scenario then just store your data in two places rather than one. You
should probably do that anyway. Two copies of data and one set of par
files will protect you very well, if not a bit of overkill. In most
cases, if you're going to use obsolete storage methods such as optical
discs, you could consider putting the data files on one disc and the par
files on another disc. Test it once in a while, (using one of the par
files, of course), to check for bit rot.

And that leaves PAR as a mechanism for storing, say, chunks
on multiple drives. RAID without a RAID. You could have
four separate hard drives, store a three chunk file, generate
a fourth PAR file, and now, if one of the four drives goes
down, the missing PAR chunk is replaced by one other chunk.

That's the right idea, but that would be a very strange implementation.
Data files don't know about chunks or par files, and there certainly
wouldn't be any reason to store each data file on a different drive.
When you create the par files, you get to decide how many chunks you
want the par files to have, but the data files don't know anything about
that. When it comes to storage, generally the par files accompany the
data files but that's not a hard requirement.

I played with PAR a bit at the time, but one of the problems
I was finding in forums, is it wasn't mathematically robust.
Some people were unable to recover damaged archives using
the additional PAR blocks they had on hand, and it was
claimed to be a "matrix problem", because the technique
involves large sparse matrices.

That sounds suspiciously like a par1 issue. By 2002 and the advent of
par2, issues like that disappeared and it's been robust ever since. I've
been using par files heavily since they were introduced, (we routinely
transfer files for remote work purposes and needed a way to verify
integrity and optionally repair damage), and I can't remember the last
time I saw a par issue. If there was one, it probably would have been in
2001, before par2 hit the scene.

On 24/06/2019 23.47, Paul wrote:
Carlos E.R. wrote:
On 24/06/2019 20.10, Paul wrote:
Carlos E.R. wrote:
On 24/06/2019 04.59, VanguardLH wrote:
Paul wrote:

[RAR] uses the old ZIP model, where decompression was free, and
compression costs money. That's why plugins are available for
decompression of RAR, which other designers can incorporate.
RARLAB's UnRAR is open source.Â* Anyone can use the free code libraries
to decompress RAR.

I always thought it presumptuous or even stupid that RARLAB charges
for
their proprietary RAR program.Â* Although claimed better, does RAR
have a
command lead in usage and is it really that much better than .zip
or .7z
that it can command a commercial price ($29 USD)?
RAR has some advantages.

It includes verification, but optionally can include repair info. That
is, a damaged archive can be repaired. This makes it good for backups.

It can easily create split archives, filling some destination media
(initially, floppies).

It can run a script (batch) when it is decompressed, which is useful
for
creating installable programs.

I did not know the price is so much. $29?

On the other hand, it does not support the full attribute set of a
Linux
filesystem. Ie, is not really multiplatform.
Many compression schemes support verification.

But last time I looked none had "repair" in a stable release considered
reliable.

To add reliability to any packaging, you can use Parchive
techniques from the outside. I don't know if anyone has
put this into, say, a multi-part archive as a means to
improving reliability.

I know about par, I have used par2 on DVDs. But not on archives.


https://en.wikipedia.org/wiki/Parchive

That was also proposed as a mechanism for enhancing
data stored on CDs or DVDs, but the problem with that
concept, is when the "error rate" gets high enough
on optical media, the laser refuses to track the spiral,
and you can't pull anything off the disc. So Parchive
for optical media is a waste of time, just because of
error characteristic (tolerates errors well, until
the laser loses lock).

Right :-(

Question. Does proper backup to DVD software exists on Windows, that do
error prevention and recovery?

And that leaves PAR as a mechanism for storing, say, chunks
on multiple drives. RAID without a RAID. You could have
four separate hard drives, store a three chunk file, generate
a fourth PAR file, and now, if one of the four drives goes
down, the missing PAR chunk is replaced by one other chunk.

I played with PAR a bit at the time, but one of the problems
I was finding in forums, is it wasn't mathematically robust.
Some people were unable to recover damaged archives using
the additional PAR blocks they had on hand, and it was
claimed to be a "matrix problem", because the technique
involves large sparse matrices.

This is a kind of error correcting code, which is likely
still in usage every day, for movie downloads. You "spray"
a server with twice as many segments as is needed to represent
a movie, the movie company does DMCA takedowns on the articles,
and if enough survive, PAR allows them to be glued together
to make an intact movie. It's something along those lines.
Maybe you have 1,2,3,29,30,31,1000,1008,1023... when the
movie is 900 long, and the movie can still be recovered
as you download more than 900 remaining segments.

Curious.

Still, par is extra work, for backups. I like the idea of rar doing the
creation of the recovery data internal, and then the repair when needed.
I mean, this is a serious advantage for rar. It is not the case in
Windows, but you know that a single byte changed in a tgz destroys it
completely. Possibly other formats, used in Windows, suffer the same
risk. Not rar.

--
Cheers, Carlos.

On 25/06/2019 17.26, Char Jackson wrote:
On Mon, 24 Jun 2019 17:47:24 -0400, Paul wrote:

To add reliability to any packaging, you can use Parchive
techniques from the outside. I don't know if anyone has
put this into, say, a multi-part archive as a means to
improving reliability.

https://en.wikipedia.org/wiki/Parchive

Well, like you said, par files are computed after the fact, when the
archive has already been created.

That was also proposed as a mechanism for enhancing
data stored on CDs or DVDs, but the problem with that
concept, is when the "error rate" gets high enough
on optical media, the laser refuses to track the spiral,
and you can't pull anything off the disc. So Parchive
for optical media is a waste of time, just because of
error characteristic (tolerates errors well, until
the laser loses lock).

I wouldn't call it a waste of time. You don't have to wait until the
read error rate gets "high enough", but if you do want to plan for that
scenario then just store your data in two places rather than one. You
should probably do that anyway. Two copies of data and one set of par
files will protect you very well, if not a bit of overkill. In most
cases, if you're going to use obsolete storage methods such as optical
discs, you could consider putting the data files on one disc and the par
files on another disc. Test it once in a while, (using one of the par
files, of course), to check for bit rot.

Backup to tape is too expensive for home usage. And there is BlueRay.

--
Cheers, Carlos.

On Wed, 26 Jun 2019 12:32:29 +0200, "Carlos E.R."
wrote:

Question. Does proper backup to DVD software exists on Windows, that do
error prevention and recovery?

That's a conundrum. Can backup software be considered "proper" if it's
backing up to DVD? Didn't backup to optical media become obsolete
something like 15+ years ago for DVD, and even longer ago for CD-R?

On Wed, 26 Jun 2019 12:35:23 +0200, "Carlos E.R."
wrote:

On 25/06/2019 17.26, Char Jackson wrote:

I wouldn't call it a waste of time. You don't have to wait until the
read error rate gets "high enough", but if you do want to plan for that
scenario then just store your data in two places rather than one. You
should probably do that anyway. Two copies of data and one set of par
files will protect you very well, if not a bit of overkill. In most
cases, if you're going to use obsolete storage methods such as optical
discs, you could consider putting the data files on one disc and the par
files on another disc. Test it once in a while, (using one of the par
files, of course), to check for bit rot.

Backup to tape is too expensive for home usage. And there is BlueRay.

Personally, I haven't seen a tape backup system in use since 1996,
whether it's in a home environment or an Enterprise environment. As for
BluRay, it's part of the optical family and appeared on the scene when
optical was already in sharp decline. BluRay wasn't able to reverse that
trend.

Char Jackson wrote:

Carlos E.R. wrote:

Backup to tape is too expensive for home usage

Personally, I haven't seen a tape backup system in use since 1996

I know of a system that backed-up daily to DC6150 tapes until about 4
years ago.

On Wed, 26 Jun 2019 18:41:07 +0100, Andy Burns
wrote:

Char Jackson wrote:

Carlos E.R. wrote:

Backup to tape is too expensive for home usage

Personally, I haven't seen a tape backup system in use since 1996

I know of a system that backed-up daily to DC6150 tapes until about 4
years ago.

A government system? They seem to usually be the last holdouts on
obsolete technologies. With their limited budgets, it's not sexy to
spend money on backup systems so other things get higher priority.

Char Jackson wrote:

Andy Burns wrote:

I know of a system that backed-up daily to DC6150 tapes until about 4
years ago.

A government system?

Yep, fire service control room system

On 26/06/2019 19.32, Char Jackson wrote:
On Wed, 26 Jun 2019 12:35:23 +0200, "Carlos E.R."
wrote:

On 25/06/2019 17.26, Char Jackson wrote:

I wouldn't call it a waste of time. You don't have to wait until the
read error rate gets "high enough", but if you do want to plan for that
scenario then just store your data in two places rather than one. You
should probably do that anyway. Two copies of data and one set of par
files will protect you very well, if not a bit of overkill. In most
cases, if you're going to use obsolete storage methods such as optical
discs, you could consider putting the data files on one disc and the par
files on another disc. Test it once in a while, (using one of the par
files, of course), to check for bit rot.

Backup to tape is too expensive for home usage. And there is BlueRay.

Personally, I haven't seen a tape backup system in use since 1996,
whether it's in a home environment or an Enterprise environment. As for
BluRay, it's part of the optical family and appeared on the scene when
optical was already in sharp decline. BluRay wasn't able to reverse that
trend.

I know enterprise people doing backup to tape often, today. They use
LTO. The units are very expensive, starting at 1500€ for an LTO-6


--
Cheers, Carlos.

On 26/06/2019 19.25, Char Jackson wrote:
On Wed, 26 Jun 2019 12:32:29 +0200, "Carlos E.R." wrote:

Question. Does proper backup to DVD software exists on Windows, that do
error prevention and recovery?

That's a conundrum. Can backup software be considered "proper" if it's
backing up to DVD? Didn't backup to optical media become obsolete
something like 15+ years ago for DVD, and even longer ago for CD-R?


Blueray.

Besides external hard disks stored in a safe box, what else is available
for home users?

--
Cheers, Carlos.

Carlos E.R. wrote:
On 26/06/2019 19.32, Char Jackson wrote:
On Wed, 26 Jun 2019 12:35:23 +0200, "Carlos E.R."
wrote:

On 25/06/2019 17.26, Char Jackson wrote:
I wouldn't call it a waste of time. You don't have to wait until the
read error rate gets "high enough", but if you do want to plan for that
scenario then just store your data in two places rather than one. You
should probably do that anyway. Two copies of data and one set of par
files will protect you very well, if not a bit of overkill. In most
cases, if you're going to use obsolete storage methods such as optical
discs, you could consider putting the data files on one disc and the par
files on another disc. Test it once in a while, (using one of the par
files, of course), to check for bit rot.
Backup to tape is too expensive for home usage. And there is BlueRay.
Personally, I haven't seen a tape backup system in use since 1996,
whether it's in a home environment or an Enterprise environment. As for
BluRay, it's part of the optical family and appeared on the scene when
optical was already in sharp decline. BluRay wasn't able to reverse that
trend.

I know enterprise people doing backup to tape often, today. They use
LTO. The units are very expensive, starting at 1500€ for an LTO-6

It's pretty simple.

Look at the price of certified tape, versus the price
of a hard drive you can trust.

And in terms of characteristics, IT people know how tape
works, what maintenance it requires (rewind once in a while,
for the cartridges, run a cleaning cassette once in a while
on the drive).

The tape wouldn't be destroyed by an EMP.

The fastest tape drive I ever heard of, was 1GB/sec, but the
cartridge for that, was a joke. Saw the sample at a conference.
I thought the cartridge was a stage prop at first, because
it was so big.

On tape farms, a layer of HDD caches are used. You run your
backups in the wee hours, with the expected "window" for your
incrementals, and the images are stored on the hard drive, until
the tape autoloader can copy the materials. With a good autoloader,
all what you need is someone labeling the materials before storage.
If the tapes had an RFID tag, or a bar code, even that could be
automated.

Paul

Carlos E.R. wrote:
On 26/06/2019 19.25, Char Jackson wrote:
On Wed, 26 Jun 2019 12:32:29 +0200, "Carlos E.R." wrote:

Question. Does proper backup to DVD software exists on Windows, that do
error prevention and recovery?
That's a conundrum. Can backup software be considered "proper" if it's
backing up to DVD? Didn't backup to optical media become obsolete
something like 15+ years ago for DVD, and even longer ago for CD-R?


Blueray.

Besides external hard disks stored in a safe box, what else is available
for home users?


The largest BluRay (write-once) is 100GB.

The media is expensive (and the pricing is based on how
many DVDs it replaces).

And you can get re-writable media.

https://www.verbatim.com/subcat/opti...blu-ray/bd-re/

And I learned something new today...

Verbatim is owned by Mitsubishi Chemical.

*******

Example of archival BD materials.

https://www.amazon.ca/Verbatim-M-Dis.../dp/B011PIJPOC

Paul

In article , Carlos E.R.
wrote:


Besides external hard disks stored in a safe box, what else is available
for home users?

a local nas, optionally (and ideally) synced to the cloud.

On Wed, 26 Jun 2019 12:32:41 -0500, Char Jackson
wrote:

On Wed, 26 Jun 2019 12:35:23 +0200, "Carlos E.R."
wrote:

On 25/06/2019 17.26, Char Jackson wrote:

I wouldn't call it a waste of time. You don't have to wait until the
read error rate gets "high enough", but if you do want to plan for that
scenario then just store your data in two places rather than one. You
should probably do that anyway. Two copies of data and one set of par
files will protect you very well, if not a bit of overkill. In most
cases, if you're going to use obsolete storage methods such as optical
discs, you could consider putting the data files on one disc and the par
files on another disc. Test it once in a while, (using one of the par
files, of course), to check for bit rot.

Backup to tape is too expensive for home usage. And there is BlueRay.

Personally, I haven't seen a tape backup system in use since 1996,
whether it's in a home environment or an Enterprise environment.


I can't remember what year it was the last time I used one, but it was
around the same time.

On Wed, 26 Jun 2019 20:08:48 +0200, "Carlos E.R."
wrote:

On 26/06/2019 19.25, Char Jackson wrote:
On Wed, 26 Jun 2019 12:32:29 +0200, "Carlos E.R." wrote:

Question. Does proper backup to DVD software exists on Windows, that do
error prevention and recovery?

That's a conundrum. Can backup software be considered "proper" if it's
backing up to DVD? Didn't backup to optical media become obsolete
something like 15+ years ago for DVD, and even longer ago for CD-R?


Blueray.

Besides external hard disks stored in a safe box, what else is available
for home users?


Web-based backup, such as Carbonite.