Long Term Data Storage

I need the best long term data storage available.

Recommendations please.

Forget about CD or DVD since I need read/write easily to one media.

I have about 500 GBytes up to 1 TByte to maintain.

Speed is not important.

Thanks !

On 01/02/2020 19:38, zernot wrote:
I need the best long term data storage available.

Recommendations please.

Forget about CD or DVD since I need read/write easily to one media.

I have about 500 GBytes up to 1 TByte to maintain.

Speed is not important.

Thanks !


External hard drive or Google Cloud Storage or Microsoft Azure or Amazon
Web Services or Alibaba or IBM Cloud or anything that turns you on.

What's your budget like for this task?

Nym-Shifter has struck again!!!!!!!!!!!!!!!!!

Path: aioe.org!news.mixmin.net!news.unit0.net!news.netfr ont.net!.POSTED.107.184.178.24!not-for-mail
From: zernot
Newsgroups: microsoft.public.windowsxp.general,alt.windows7.ge neral
Subject: Long Term Data Storage
Date: Sat, 1 Feb 2020 11:38:32 -0800
Organization: Netfront http://www.netfront.net/
Message-ID:
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sat, 1 Feb 2020 19:48:47 -0000 (UTC)
Injection-Info: adenine.netfront.net; posting-host="107.184.178.24";
logging-data="16868"; "
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101
Firefox/49.0 SeaMonkey/2.46
X-Mozilla-News-Host: news://freenews.netfront.net:119
Xref: aioe.org microsoft.public.windowsxp.general:155579 alt.windows7.general:48359



--
With over 1.2 billion devices now running Windows 10, customer
satisfaction is higher than any previous version of windows.

zernot wrote:
I need the best long term data storage available.

Recommendations please.

Forget about CD or DVD since I need read/write easily to one media.

I have about 500 GBytes up to 1 TByte to maintain.

Speed is not important.

One machine, no redundancy, another sata hdd. Network, no redundancy
rig a RPi4 plus USB3 sata hdd NAS. Network redundancy, rpi4 + 2 usb3 satas.

--
Mike Easter

Is there code available for the RPi4 to do the LAN to USB drives or is
that standard internal code ?

Links please.


Mike Easter wrote:
zernot wrote:
I need the best long term data storage available.

Recommendations please.

Forget about CD or DVD since I need read/write easily to one media.

I have about 500 GBytes up to 1 TByte to maintain.

Speed is not important.

One machine, no redundancy, another sata hdd. Network, no redundancy
rig a RPi4 plus USB3 sata hdd NAS. Network redundancy, rpi4 + 2 usb3
satas.

On 2020-02-01, zernot wrote:
I need the best long term data storage available.

Stone tablets. Not very fast, and limited storage density, but they have
been known to survive thousands of years under extreme conditions.

--
-----------------------------------------------------------------------------
Roger Blake (Posts from Google Groups killfiled due to excess spam.)

The US Census vs. privacy -- http://censusfacts.info
Don't talk to cops! -- http://www.DontTalkToCops.com
Badges don't grant extra rights -- http://www.CopBlock.org
-----------------------------------------------------------------------------

zernot wrote:
Is there code available for the RPi4 to do the LAN to USB drives or is
that standard internal code ?

There are lots of articles about doing it and doing it different ways
all the way from people creating enclosures for the rpi and the drives
using 3D printer to using commercially available hat for the rpi for
multiple sata drives.

I'll pick an example of an easy one that doesn't mention the rpi4 which
I think would be the easiest because it already has the usb3 if that is
what you do for the hdd.

Since this is in a Win group, I'll pick one which (also) uses Win
putty/ssh for the headless operation of the rpi. The article also has
links for doing it w/ linux as well.

https://howtoraspberrypi.com/create-...-pi-and-samba/
Create a NAS with your Raspberry Pi and Samba

That article assumes you are ssh connected, which is described he
https://howtoraspberrypi.com/ssh-ras...emote-control/ Learn how
to use SSH to remote control your Raspberry Pi

The first link above uses Samba for the NAS: "Creating the NAS server
with Samba"

--
Mike Easter

Thanks Mike.

Ordered RPi 4B kit and receiving it tomorrow.

This will be a Windows to LAN to RPi to USB.

Limitations ?

Where else to go for info. i.e. newsgrops etc.

Thanks again.


Mike Easter wrote:
zernot wrote:
Is there code available for the RPi4 to do the LAN to USB drives or is
that standard internal code ?

There are lots of articles about doing it and doing it different ways
all the way from people creating enclosures for the rpi and the drives
using 3D printer to using commercially available hat for the rpi for
multiple sata drives.

I'll pick an example of an easy one that doesn't mention the rpi4 which
I think would be the easiest because it already has the usb3 if that is
what you do for the hdd.

Since this is in a Win group, I'll pick one which (also) uses Win
putty/ssh for the headless operation of the rpi. The article also has
links for doing it w/ linux as well.

https://howtoraspberrypi.com/create-...-pi-and-samba/
Create a NAS with your Raspberry Pi and Samba

That article assumes you are ssh connected, which is described he
https://howtoraspberrypi.com/ssh-ras...emote-control/ Learn how
to use SSH to remote control your Raspberry Pi

The first link above uses Samba for the NAS: "Creating the NAS server
with Samba"

zernot wrote:
Thanks Mike.

YW.

Ordered RPi 4B kit and receiving it tomorrow.

This will be a Windows to LAN to RPi to USB.

Limitations ?

Well; it depends on how easily you can adapt to the linux system
'engaged' w/ Win.

Where else to go for info.Â* i.e. newsgrops etc.

There's a rpi group which has some pretty knowledgeable folks, some are
very hardware oriented. For this kind of project, there's a lot of
articles online you can find w/ a search. Some people have made some
'serious' NAS w/ RPi base, others have simpler systems.

The rpi group comp.sys.raspberry-pi

My own experience w/ RPi is just a 'simple' RPi3B which has its own
monitor kb mouse, not NAS and I'm familiar w/ using llnux systems but
not remotely, but networked w/ Win systems in the network. And I read
about stuff like this, as opposed to 'doing'.


--
Mike Easter

zernot wrote:

Is there code available for the RPi4 to do the LAN to USB drives or is
that standard internal code ?

Links please.


If you're asking questions like this, an "appliance" is
probably a better answer.

RPi is a bit deceptive, in that the base computer board is
cheap, but most people will want a housing, power supply,
and other do-dads to go with it. To make a complete product.
It would end up almost as expensive as one of these.

Synology DS120j BYOD (Bring your own disk) NAS (Network Attached Storage) $100

https://www.newegg.com/synology-ds12...82E16822108637

One 3.5" drive bay
Two USB3 connectors for external devices

That's about as cheap as they get today.

A two bay unit, needs a bit more electricity, but I don't
know if it justifies a $167 price (when it has no disks inside
at purchase).

Synology 2 bay NAS DiskStation DS218j (Diskless) $167
https://www.newegg.com/synology-ds21...82E16822108688

Synology Hybrid RAID (SHR), Basic, JBOD, RAID 0, RAID 1
Two 3.5" drive bays
Two USB3 connectors for external devices

These usually support fairly high capacity disks. You want to
check the capacity figure, before you buy.

*******

You can occasionally find a review.

https://www.smallnetbuilder.com/nas/...tions-reviewed

With two disk bays, you can place two 1TB drives and operate RAID 1.
If one disk dies, the theory is the second disk continues to operate.
RAID 1 mode is "Mirror" mode, so the disks are supposed to remain
identical. To achieve that, you might want to power it from a UPS,
and do orderly shutdowns when the power fails and the unit remains
powered by the UPS. Externally, it looks like a single 1TB volume,
but it's stored on two identical disks. If one disk is fried, the
second disk still has your data.

I find the packaging for my home projects costs a fortune,
which is why I'd want to do the maths on an RPi project
carefully before going that route.

On the minus side, the software and interfaces that come
with NAS products, aren't everyones cup of tea. So there's
that.

I'm a little too cheap for this sort of stuff, and
you won't catch me buying stuff like this. I just
plug a disk into the machine and do whatever
is necessary that way. The disks on my machines load from
the side. My typing machine has a "rack" of sorts
next to the machine, and drives go into that thing,
rather than into 5.25" bays in the main case. So when I
mention putting disks in the machine, I'm sure some people
think I'm struggling with 5.25" holes on the front of
the machine, whereas I'm just sliding Antec trays
into the side of the machine. That's a fair bit easier.

And I have plenty of redundancy and way more drives
than it is wise to own. I have at least two dozen drives now.
The problem with that many drives, is never having a full
inventory of what's on them :-) I do have an "Inventory" folder,
but it's completely out of date.

And what you'd be using on the NAS, is SMB/CIFS (Windows File Sharing).
If you've had trouble with that, then that factors into your
decision to go this route. Some people never get Windows File Sharing
working worth a darn, and part of the reason is their having
an excessively complicated network setup. (They don't know how
their networking works, or doesn't work.) I can tell you when I
first started file sharing with the OSes more modern than
WinXP, there were times I wanted to throw stuff out the window.
But today, it all works. Even if I have to use IP addresses to
address stuff part of the time (Linux side).

An alternative with the NAS, is FTP. It also has FTP protocol,
suited to usage on the local LAN only. If you're having trouble
getting full speed operation, and have a really big file to transfer,
you can do it with FTP instead, and the transfer will go faster.
I once transferred an entire hard drive using FTP :-)

Paul

zernot wrote:

I need the best long term data storage available. Forget about CD or
DVD since I need read/write easily to one media. I have about 500
GBytes up to 1 TByte to maintain. Speed is not important.

100GB is the largest you can go with optical media (BD 4-layer);
however, writable optical media uses chemical phase change to emulate
pits burned in my the commercial "printing" process, and the chemical
phase change wanes over time. There are optical discs rated for
archival storage of 100 years, but you have to store them out of
sunlight and in a cool place. There are plans to increase optical
capacity, but they'll be very pricey when they appear. For example:

https://www.cdrom2go.com/blog/the-fu...d-optical-disc

A problem with optical media is its capacity decreases with subsequent
write sessions. When you write data to the optical disc, you're likely
to close the session to prevent accidental deletion of the file(s).
When you next write to the disc, you start a new session. The old
session remains on the disc. That lets you traverse through old
versions of the same file(s) by walking back through the sessions on the
disc. However, eventually there is no more free space or not enough
remaining free space in a new session for the size of a file you are
trying to write to it. That means having to erase the disc (lose ALL
files) and start a new session (where you hope writing the old files on
the newly erased disc won't have a write failure, and you lose all those
old file versions). That's assuming you're using rewritable media. If
you're using write-one media, you can't erase it, the old sessions stay
on the disc, and the disc's capacity keeps decreasing.

Everyone weaned on computers in the last decade tends to forget or
doesn't even know about tape storage. HDDs have a MTBF of 9 years.
Tape is 46 years. Google uses tape for archival storage. There are
large capacity cartridges and drives to handle them, but they're
probably outside your budget.

https://www.ironmountain.com/resourc...hive-solutions

You never mention what hardware protocol you want to use for the backup
media. SATA, USB, something else? And if you are going to keep around
the old computer to make sure that the hardware protocol is still
available after decades of archival storage (since computers change and
old hardware protocols fade away).

Why not use a USB-attached 1, 2, or 4 TB external HDD? Just remember
that magnetic media suffers from dipole stress which means bit
differentiation becomes more difficult; i.e., the bits will fade.
Magnetic media needs refreshing (the data must be rewritten) every few
years (figure on a 5-year refresh cycle for stored -- not used -- HDDs
unless you rotate the HDDs sufficiently that the data doesn't survive
that long). HDDs are mechanical devices, so they suffer from mechanical
stress. That's why optical media is often used, because if the drive
goes bad the media is still okay, and you get a replacement drive (if
still available) to read the old media. If the mechanicals in an HDD go
bad, you either lose that data (but you should have duplicates, anyway)
or you pay a recovery lab to lift the data off the platters. If you're
archiving to HDDs, you don't use those for backups. Backup and
archiving are not the same thing. Backups are for disaster recovery
(data or hardware). Archives are for data recovery.

SSDs are not as cost effective as HDDs, especially for archival storage.
Also, SSDs are destructive: the more you write on them, to close you
approach their maximum write cycle count. They self-destruct as they
are written. If you write just once, they'll last a very long time.
The more you write to them, the shorter their remaining lifespan. They
don't suffer the failure from mechanicals as do HDDs. Flash drives are
also self-destructive writable devices, and the cheapies are not safe
for archival storage.

No matter what media you choose, you'll need to retain a working
computer that provides you with all the old hardware protocols needed to
support that media. SATA and USB are common now. Who knows what will
get used a decade from now, and after that change it could be another
decade when you can no longer get the hardware to support your old
archival media. You need an archival computer that is used only to make
sure that archival media remains usable. That archival computer gets
stored, not used as a workstation for everyday use. After all, sometime
in the future, you will need that archived computer to read your archive
media along with the programs to use the recovered data.

You aren't planning to have just one copy for archival storage, right?
And those copies won't be stored at the same location, right? And you
won't be using them for backups, right?

You're just storing data files onto the archival media, right? You
don't need to keep saving duplicate copies of your programs into your
archive. You only need 1 to 3 copies of your installation media
depending on how many places you are going to store your archives
(should be more than 1, like at work and home and somewhere else to
protect against fire, flood, tornado, etc).

https://blog.macrium.com/what-type-o...g-e3ef2eb75152

Without a budget, no one knows what to suggest to you, plus your
duration for archival is unknown.

In message , VanguardLH
writes:
zernot wrote:

I need the best long term data storage available. Forget about CD or

Punched plastic tape perhaps - but you'd need the square miles of
building to store them in.

DVD since I need read/write easily to one media. I have about 500

("Read/write easily" and "archive storage" are probably in the end
conflicting rquirements.)

GBytes up to 1 TByte to maintain. Speed is not important.

(You say that, but I presume within limits; consider my punched tape -
even with the 60-70 MPH that I think they got up to with Colossus, I
suspect 500G-1T would still take a while ...)

100GB is the largest you can go with optical media (BD 4-layer);
however, writable optical media uses chemical phase change to emulate
pits burned in my the commercial "printing" process, and the chemical
phase change wanes over time. There are optical discs rated for
archival storage of 100 years, but you have to store them out of
sunlight and in a cool place. There are plans to increase optical
capacity, but they'll be very pricey when they appear. For example:

https://www.cdrom2go.com/blog/the-fu...he-terabyte-si
zed-optical-disc

A problem with optical media is its capacity decreases with subsequent

It's capacity for _additional_ data, yes.

write sessions. When you write data to the optical disc, you're likely
to close the session to prevent accidental deletion of the file(s).
When you next write to the disc, you start a new session. The old
session remains on the disc. That lets you traverse through old
versions of the same file(s) by walking back through the sessions on the
disc. However, eventually there is no more free space or not enough
remaining free space in a new session for the size of a file you are
trying to write to it. That means having to erase the disc (lose ALL
files) and start a new session (where you hope writing the old files on
the newly erased disc won't have a write failure, and you lose all those
old file versions). That's assuming you're using rewritable media. If
you're using write-one media, you can't erase it, the old sessions stay
on the disc, and the disc's capacity
for _new_ data
keeps decreasing.

As an aside: you said "writable optical media uses chemical phase change
.... and the chemical phase change wanes over time"; I think you were
describing _re_-writable media. I think write-on[c]e media really does
"burn", hence the expression; any thoughts on the decay of that? (From
my experience, the _re_writable stuff's reliability lifetime is _very_
short, but that may be because I used what I think was sometimes called
packet format, making the CD-RW into a giant floppy [gave IIRR a reduced
capacity, I think about 500M on a 700M disc, but you could erase files
and the space became available again].) For -R discs, I've some that are
still usable after over a decade, but some that are certainly not. (And
optical disc formats - both CD and DVD - seem to have the unusual
property that they really screw up the OS; for example, I have a couple
of discs from a standalone DVD [video] recorder, which were never
"closed" before the machine died; trying to read them on a PC not only
results in a a lack of success, as you might expect, but really screws
up the OS, such that to get control back, I at the very least have to
eject the discs. Similarly with the CD-Rs that have failed: trying to
read them makes the OS behave - hard to pin down, best description is as
if it has a bad case of 'flu!)

Everyone weaned on computers in the last decade tends to forget or
doesn't even know about tape storage. HDDs have a MTBF of 9 years.
Tape is 46 years. Google uses tape for archival storage. There are
large capacity cartridges and drives to handle them, but they're
probably outside your budget.

https://www.ironmountain.com/resourc...best-long-term
-data-archive-solutions

Tape is sequential, and slow - and the drives _can_ be complex
mechanically; I'd believe your claim of 46 years (though of course it
must have been derived by extrapolation) for the tapes, but not the
drives - complex loading mechanisms, perishing belts ...
(Nevertheless, I have lived through several periods when tape was by far
the cheapest - and I could believe the best volume-wise - medium for
bulk storage, and if you tell me we're in such a period again, I'm happy
to believe you.)

You never mention what hardware protocol you want to use for the backup
media. SATA, USB, something else? And if you are going to keep around
the old computer to make sure that the hardware protocol is still
available after decades of archival storage (since computers change and
old hardware protocols fade away).

Why not use a USB-attached 1, 2, or 4 TB external HDD? Just remember
that magnetic media suffers from dipole stress which means bit
differentiation becomes more difficult; i.e., the bits will fade.
Magnetic media needs refreshing (the data must be rewritten) every few
years (figure on a 5-year refresh cycle for stored -- not used -- HDDs

Probably simplest to just do a copy from one drive to another, but do
you know of any software that will do a refresh on a single drive - i.
e. read and definitely write it back [not just not doing the write as it
"knows" it hasn't changed]? (Thinking about it, probably having two
equal-size partitions and just copying the structure back and forth
between them would do it.)

unless you rotate the HDDs sufficiently that the data doesn't survive
that long). HDDs are mechanical devices, so they suffer from mechanical
stress. That's why optical media is often used, because if the drive
goes bad the media is still okay, and you get a replacement drive (if
still available) to read the old media. If the mechanicals in an HDD go

Or your tape drive, of course. Though I suspect the drive availability
might be worse.

bad, you either lose that data (but you should have duplicates, anyway)
or you pay a recovery lab to lift the data off the platters. If you're
archiving to HDDs, you don't use those for backups. Backup and
archiving are not the same thing. Backups are for disaster recovery
(data or hardware). Archives are for data recovery.

SSDs are not as cost effective as HDDs, especially for archival storage.
Also, SSDs are destructive: the more you write on them, to close you
["the closer" (-:]
approach their maximum write cycle count. They self-destruct as they
are written. If you write just once, they'll last a very long time.

Interesting thought. Yes, probably a pretty good archival medium -
though very expensive at the moment. Though have they been around long
enough for true information on their _static_ bit-rot to have been
obtained? (Especially the two or three bits per cell - multilevel? -
types, that Paul's always being depressed about the march of.)
[]
No matter what media you choose, you'll need to retain a working
computer that provides you with all the old hardware protocols needed to

And transitional hardware - how are you going to get the data out! If I
had an old PC, with working 5¼" floppy drives, and some floppies that
hadn't faded, still how would I get the data out: probably through its
serial port - if I could find a modern machine with a serial port! Maybe
that's what you mean by:

support that media. SATA and USB are common now. Who knows what will
get used a decade from now, and after that change it could be another
decade when you can no longer get the hardware to support your old
archival media. You need an archival computer that is used only to make
sure that archival media remains usable. That archival computer gets
stored, not used as a workstation for everyday use. After all, sometime
in the future, you will need that archived computer to read your archive
media along with the programs to use the recovered data.

So arguably, your (5-10 year?) refresh cycle shouldn't just refresh the
data on the old media/hardware, it should move it onto more up-to-date
media. Not as onerous as it sounds, as the capacity of storage media
seems to shoot up faster than Moore's law - though still _fairly_
onerous, as the copying/moving will be limited by the speed of the _old_
hardware each time.

(Even more of a problem in the analogue world, for two reasons:
equipment to _play_ the old material [e. g. obscure videotape formats -
or even: when did you last see a film projector, or reel-to-reel tape
machine?]; and, analogue copying degrades each time. [OK, digitise. But
that assumes you're happy you've got all that can be got: for example
much film has been digitised in SD, HD ... but may contain information
that makes it worth doing in UHD, or something not even thought of yet.
And extraction techniques - things are always being invented, such as
scanning film using IR or UV to detect and correct for scratches.])

You aren't planning to have just one copy for archival storage, right?
And those copies won't be stored at the same location, right? And you
won't be using them for backups, right?

You're just storing data files onto the archival media, right? You
don't need to keep saving duplicate copies of your programs into your
archive. You only need 1 to 3 copies of your installation media
depending on how many places you are going to store your archives
(should be more than 1, like at work and home and somewhere else to
protect against fire, flood, tornado, etc).

Assuming there'll still be something - even a VM - onto which it _can_
be installed (-:.

https://blog.macrium.com/what-type-o...-i-be-using-e3
ef2eb75152

Without a budget, no one knows what to suggest to you, plus your
duration for archival is unknown.

True. Some hints on both - plus the type of data, and its purpose -
wouldn't go amiss.
--
J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf

The modern world so often thinks that the way to relax is by doing absolutely
nothing, and I've never really understood that.
Nigella Lawson in RT 2015/10/31-11/6

"J. P. Gilliver (John)" wrote:

VanguardLH WROTE:

It's capacity for _additional_ data, yes.

Nope, for each *session*. You can save the SAME file in each session.
You can save a different version of the file in each session. You can
include a file in one session but not in another session. For example,
you could delete files or exclude them in a /new/ session: the file is
gone in the new session, perhaps to sync with the current state of the
file system, so you are reducing data in the new session, but the new
session still sucks up space on the optical media. You could keep
creating new sessions to repeatedly store the same unchanged file in
each session and eventually eat up all the space on the disc. You
haven't saved anything "new".

Typically only the last session is accessible. The old sessions are
still on the disc, but are hidden. Being hidden does not erase them nor
return storage space to the disc. Folks will complain they cannot write
a 10KB file to a 4GB DVD because they don't know all those old sessions
are still occupying space on the disc.

ISObuster and other disc tools will show you any retained old sessions
still on the disc.

https://www.isobuster.com/multisessions.php

for _new_ data

Yep, in the /new/ session (with the older sessions being hidden). "New"
just means another copy of the file whether it got changed or not from
the copy of the file saved in an old session. So it could be "old" data
by having the same file without change duplicated in the new session.

As an aside: you said "writable optical media uses chemical phase change
... and the chemical phase change wanes over time"; I think you were
describing _re_-writable media.

Nope, write-once or re-writable discs both use a chemical phase change
to emulate a pit. You don't have the glass plates with pits and press
to make duplicates from a template to have a commercial-quality disc
that has real pits. All you have are the burner drives that cause a
chemical phase change using a laser. Your discs are "burned" (a laser
causes a phase change in the chemicals), not pressed against a glass
master.

I think write-on[c]e media really does "burn", hence the expression;
any thoughts on the decay of that?

"Burn" just means using a laser to alter a chemical dye layer. More
accurate would be "heated chemical phase change". Which do you think
marketers are likely to use when selling their wares to the vast
majority of boobs using computers?

Think about it: if the laser in a disc drive were to actually burn the
disc, you'd smell it plus there would be bubbling or warping caused by
damaging the recording layer. Discs aren't hollow.

Marketers aren't about advertising the truth. They advertise to sell.
Depends on the market to which they sell regarding the expertise of that
customer market. They can't be too technical nor accurate for computer
wares.

Tape is sequential, and slow - and the drives _can_ be complex
mechanically; I'd believe your claim of 46 years (though of course it
must have been derived by extrapolation) for the tapes, but not the
drives - complex loading mechanisms, perishing belts ...

It's the archival storage time of the media to which I referred. As
with tapes, discs, or hard disks, you are reliant on the workings of a
mechanical device continuing to work for years or decades. The media
survives. You simply replace a malfunctioning drive when you use
removable media for archival storage. With tapes and discs, replacing
the drive is easy. With HDDs, you need to pay a lab to either move the
platters to a replacement disk housing or get them to lift the data off
the platter to copy onto a new hard disk. Did you forget that hard
disks are also mechanical devices?

Archival storage duration isn't measured by the lifespan of the
mechanical drive, but by how long the media retains the data.

Probably simplest to just do a copy from one drive to another, but do
you know of any software that will do a refresh on a single drive - i.
e. read and definitely write it back [not just not doing the write as it
"knows" it hasn't changed]? (Thinking about it, probably having two
equal-size partitions and just copying the structure back and forth
between them would do it.)

SpinRite and HDD Regenerator are just a couple examples of data recovery
software that will test free space to make sure it is reliable for
landing of a copy of data, test the used sectors to make the read will
be reliable, and then copy from the old sector to the new one. The
write with move performs a refresh. Alternatively, if there isn't
enough free space for the entire file, memory could be used for holding
a sector of a file, and then test and refresh the old sector by writing
the file back to it. I used to have HD Sentinel and, as I recall, its
payware versions had disk diagnostics that included data refresh. I've
heard of but never used DiskFresh.

Copying from one drive to another require another drive. What if all
the onboard controller slots are used, so you cannot add another
internal drive. Yep, you could use a USB-attached drive, but remember
the data was wanted on the internal drive, so you wouldn't be copying
once but twice: once from the internal drive to the external one, and
again to copy from the external drive back to the internal drive. The
problem with just a copy is that you assume the formatting results in a
reliable copy. The tools used for refresh ensure the target to where
the copy gets put is tested beforehand, and are more thorough than
running "chkdsk /r" that supposedly tests the sectors before using them.

I never had to refresh an HDD because those didn't last long enough in
my setup to need a refresh to ensure data integrity. Typically the HDDs
get replaced in about 5 years to get bigger (larger capacity) and faster
ones (the drive was faster or move to a faster hardware protocol). I
have had to refresh floppies where dipole stress is a worse problem.
The data that wasn't readable not only became readable but it got
refreshed to realign the dipoles for maximum signal strength to improve
differentiation of bits on the media.

Anyone who archives files but never refreshes the magnetic media is
asking for corrupted or unreadable files sometime in the future when
they desparately need those files. Mortgage companies are required to
keep records for 40 years.

No matter what media you choose, you'll need to retain a working
computer that provides you with all the old hardware protocols needed to

And transitional hardware - how are you going to get the data out! If I
had an old PC, with working 5¼" floppy drives, and some floppies that
hadn't faded, still how would I get the data out: probably through its
serial port - if I could find a modern machine with a serial port! Maybe
that's what you mean by:

By using that archived computer that had the 5.25" and 3.5" floppy
drives you used to create the archived floppies. You need to reuse
those same drives when you restore from the archived floppies due to
alignment problems. Never ran into the problem where you wrote a floppy
disk on one computer but couldn't read it in a floppy drive on another
computer? That was due to misalignment of one of the drives.
Eventually they went to laser markings on the floppy (actually burned
into the media; "LS" means laser servo) to eliminate the alignment
problems (https://en.wikipedia.org/wiki/SuperDisk). Remember the old
Iomega Zip drives that use the "floptical" Zip disks? Those were LS
drives. However, by then the floppy format was undersized and too slow
compared to optical media, and even Iomega Zip media faded away (I have
one remaining computer with a Zip drive along with a couple storage
cases full of Zip disk, but everything on them has been copied
elsewhere, so they're awaiting the discard of the old computer with the
Zip drive).

So arguably, your (5-10 year?) refresh cycle shouldn't just refresh the
data on the old media/hardware, it should move it onto more up-to-date
media.

That's an expensive proposition. Why buy new discs, tapes, of HDDs when
the old media is perfectly still usable? The reason the refresh
software is still a viable and cost effective solution to insurance of
magnetic archival of data is that the hardware gets reused, not
replaced.

In message , VanguardLH
writes:
"J. P. Gilliver (John)" wrote:

VanguardLH WROTE:

It's capacity for _additional_ data, yes.

Nope, for each *session*. You can save the SAME file in each session.
You can save a different version of the file in each session. You can
include a file in one session but not in another session. For example,
you could delete files or exclude them in a /new/ session: the file is
gone in the new session, perhaps to sync with the current state of the
file system, so you are reducing data in the new session, but the new
session still sucks up space on the optical media. You could keep
creating new sessions to repeatedly store the same unchanged file in
each session and eventually eat up all the space on the disc. You
haven't saved anything "new".

Typically only the last session is accessible.

That's the important line. Yes, if you're just using remaining space for
the entire new session, including [new copies of] files that haven't
changed, of course it will be as you say. And yes, I know that's the
default way these discs (well, the software that uses them) work.
(Though there was what I think was called "packet" for RW media, that
_did_ make the disc work like a giant floppy, in other words space from
a deleted file _did_ become immediately available - but you had to have
the "packet driver" installed on all machines. IIRR, the two main makers
as the time - Roxio/adaptec and Nero, I think - had their own systems [I
think adaptec's was called DirectCD, and Nero's InCD]; I am not sure
they were compatible. For some reason they gave a reduction in capacity,
to IIRR 5xx MB on a 700 MB CDRW. I used to use them for transferring
things between computers, much as I might a USB stick now; I didn't find
them all that reliable [had to reformat them after not very long], and
certainly wouldn't archive to them, or to RW using even the normal burn
method.)

The old sessions are
still on the disc, but are hidden. Being hidden does not erase them nor
return storage space to the disc. Folks will complain they cannot write
a 10KB file to a 4GB DVD because they don't know all those old sessions
are still occupying space on the disc.

ISObuster and other disc tools will show you any retained old sessions
still on the disc.

https://www.isobuster.com/multisessions.php

Ah. $39.95 - quite reasonable, but I don't currently have a need for it.
(I have vague memories of having used something freeware in the past
that could see old sessions, but I'm not sure about that.)

for _new_ data

Yep, in the /new/ session (with the older sessions being hidden). "New"
just means another copy of the file whether it got changed or not from
the copy of the file saved in an old session. So it could be "old" data
by having the same file without change duplicated in the new session.

Seems odd that the industry didn't make a version that defaulted to
letting you access existing copies of unchanged files; well, I suppose
not so odd in that they want to sell more and more blanks, but as the
drive manufacturers weren't always the same companies as the blank
manufacturers, you'd have thought one would have broken ranks ...

As an aside: you said "writable optical media uses chemical phase change
... and the chemical phase change wanes over time"; I think you were
describing _re_-writable media.

Nope, write-once or re-writable discs both use a chemical phase change
to emulate a pit. You don't have the glass plates with pits and press
to make duplicates from a template to have a commercial-quality disc
that has real pits. All you have are the burner drives that cause a
chemical phase change using a laser. Your discs are "burned" (a laser
causes a phase change in the chemicals), not pressed against a glass
master.

I realise they don't have physical pits, but all that matters is that
the beam either reflects or doesn't; in a pressed CD it's because the
pits are a quarter wavelength (or appropriate multiple thereof) deep and
reflections from them cancel reflections from the surface beside the
pits or doesn't where there's no pit, in R/RW it's because the material
is opaque or isn't.

I think write-on[c]e media really does "burn", hence the expression;
any thoughts on the decay of that?

"Burn" just means using a laser to alter a chemical dye layer. More
accurate would be "heated chemical phase change". Which do you think
marketers are likely to use when selling their wares to the vast
majority of boobs using computers?

Think about it: if the laser in a disc drive were to actually burn the
disc, you'd smell it plus there would be bubbling or warping caused by
damaging the recording layer. Discs aren't hollow.

Not at the microscopic size involved, surely (either smell or warping) -
you'd only need to create a tiny black spot inside the plastic. (As is
done to make the artwork in various awards - the sort given at awards
ceremonies - which are a block of clear plastic with shapes inside.)

Marketers aren't about advertising the truth. They advertise to sell.
Depends on the market to which they sell regarding the expertise of that
customer market. They can't be too technical nor accurate for computer
wares.

However, I'm willing to believe you're right, and it's just a "chemical
phase change" for both RW and R. (Though it could be argued that
blackening by burning is just a chemical change of course!)

I think _part_ of the decay mechanism for R discs isn't just the "burnt"
bits reverting to non-burnt (if that does indeed happen), but also the
non-burnt parts may darken with time (especially if not kept out of the
light), thus reducing the difference between the two, which is after all
what's important.

Tape is sequential, and slow - and the drives _can_ be complex
mechanically; I'd believe your claim of 46 years (though of course it
must have been derived by extrapolation) for the tapes, but not the
drives - complex loading mechanisms, perishing belts ...

It's the archival storage time of the media to which I referred. As
with tapes, discs, or hard disks, you are reliant on the workings of a
mechanical device continuing to work for years or decades. The media
survives. You simply replace a malfunctioning drive when you use
removable media for archival storage. With tapes and discs, replacing
the drive is easy. With HDDs, you need to pay a lab to either move the
platters to a replacement disk housing or get them to lift the data off
the platter to copy onto a new hard disk. Did you forget that hard
disks are also mechanical devices?

Archival storage duration isn't measured by the lifespan of the
mechanical drive, but by how long the media retains the data.

In academic theory, yes; for practical use, there's no point in having a
tape that you are absolutely sure still has your wanted bits on it, if
you don't have a reader that still works (and can't get one)! I haven't
looked into data tape drives for ages (probably decades), but I assume
that, like other media, they've developed over the years to handle new
cartridge designs to handle higher capacities: _are_ they
backwards-compatible, in other words if you buy a new tape drive today,
will it still be able to read a cartridge from 20 or 30 years ago, as
well as today's? (It's possible they are - after all, a DVD drive can
_usually_ still read a CD [not sure about a blu-ray drive] - but I don't
know, as tape is a more specialised area.)

Probably simplest to just do a copy from one drive to another, but do
you know of any software that will do a refresh on a single drive - i.
e. read and definitely write it back [not just not doing the write as it
"knows" it hasn't changed]? (Thinking about it, probably having two
equal-size partitions and just copying the structure back and forth
between them would do it.)

SpinRite and HDD Regenerator are just a couple examples of data recovery
software that will test free space to make sure it is reliable for
landing of a copy of data, test the used sectors to make the read will
be reliable, and then copy from the old sector to the new one. The
write with move performs a refresh. Alternatively, if there isn't
enough free space for the entire file, memory could be used for holding
a sector of a file, and then test and refresh the old sector by writing
the file back to it. I used to have HD Sentinel and, as I recall, its
payware versions had disk diagnostics that included data refresh. I've
heard of but never used DiskFresh.

I wrote the above paragraph thinking that just reading doesn't refresh,
you need to rewrite, but even as I was writing it I realised that
there's no strong reason that it has to be rewritten _in the same place
on the disc_, so as long as it is actually written not just read, the
refresh process has happened. (Well, write _with verify_ of course.)
[]
reliable copy. The tools used for refresh ensure the target to where
the copy gets put is tested beforehand, and are more thorough than
running "chkdsk /r" that supposedly tests the sectors before using them.

I've often seen people disparaging chkdsk /r and other similar; I'm not
saying anyone is wrong to disparage it, but I've also yet to see any
actual _proof_ that it actually lies, rather than just misleads. (Though
the in-drive-electronics swapping etc. may enter into that discussion.)
[]
Anyone who archives files but never refreshes the magnetic media is
asking for corrupted or unreadable files sometime in the future when

Agreed.

they desparately need those files. Mortgage companies are required to
keep records for 40 years.

[In your country. I don't know for here (UK) - might be more, or less.]

No matter what media you choose, you'll need to retain a working
computer that provides you with all the old hardware protocols needed to

And transitional hardware - how are you going to get the data out! If I
had an old PC, with working 5¼" floppy drives, and some floppies that
hadn't faded, still how would I get the data out: probably through its
serial port - if I could find a modern machine with a serial port! Maybe
that's what you mean by:

By using that archived computer that had the 5.25" and 3.5" floppy
drives you used to create the archived floppies. You need to reuse
those same drives when you restore from the archived floppies due to
alignment problems. Never ran into the problem where you wrote a floppy

Ah, we're talking at cross-purposes here. (I agree with you about
alignment issues.)
[]
So arguably, your (5-10 year?) refresh cycle shouldn't just refresh the
data on the old media/hardware, it should move it onto more up-to-date
media.

That's an expensive proposition. Why buy new discs, tapes, of HDDs when
the old media is perfectly still usable? The reason the refresh
software is still a viable and cost effective solution to insurance of
magnetic archival of data is that the hardware gets reused, not
replaced.

It depends _why_ you need to access the old data. If it's to do
something that can only be _done_ on the old computer, then we're into a
completely different ballgame - you need to look at not only data
archiving, but hardware retention/maintenance (and arguably duplication
for redundancy), which is a whole new vale of tears! If you only need to
access the data for say legal reasons, albeit maybe "running" it on a VM
simulation of the old computer, then it doesn't matter what it's on, and
arguably the more modern the better. Your mortgage companies, for
example, I presume (hope!) they copied anything they had on punched
cards, paper tape, or even half-inch tape, onto something more modern,
some while ago.

(And the expense may not be the case: maintaining _really_ old media -
_and the hardware needed to access it_ [with redundant duplicates] - can
be expensive, and may even not be _possible_ as those with knowledge of
it retire, and even die.)
--
J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf

Society has the right to punish wrongdoing; it doesn't have the right to make
punishment a form of entertainment. This is where things have gone wrong:
humiliating other people has become both a blood sport and a narcotic.
- Joe Queenan, RT 2015/6/27-7/3

"J. P. Gilliver (John)" wrote:

Ah. $39.95 [for IsoBuster].

Use the free version. During installation, opt for the free version.
Yep, that means some features are disabled.

https://www.isobuster.com/license-models.php

There are certainly alternatives to IsoBuster, but that's the one that I
know about. I haven't installed it in my latest computer build, but
that would be what I'd use if I needed its features.

Seems odd that the industry didn't make a version that defaulted to
letting you access existing copies of unchanged files; well, I suppose
not so odd in that they want to sell more and more blanks, but as the
drive manufacturers weren't always the same companies as the blank
manufacturers, you'd have thought one would have broken ranks ...

You can certainly NOT close a session. You could then keep "burning"
more files to the same session. Alternatively, you can create a multi-
session disc: if you copy the same file (changed or not), the old file
remains but the TOC will point to the last saved file by the same name
and path.

https://www.lifewire.com/how-to-crea...n-disc-2438838

The OP was interested in long term storage which makes it appear the OP
wants archival storage, and you close those sessions to protect the
files against accidental deletion. You're not supposed to keep
modifying archival media. That's why I said backups are not the same as
archiving.

I realise they don't have physical pits, but all that matters is that
the beam either reflects or doesn't;

It is because it is chemical for why cheap media is not applicable for
archival use. Remember the OP is looking for long time storage (but
didn't say how long). Yep, you want the laser reaction to continue to
work, but that means you are relying on the integrity of the media. For
backups that maybe have a retention of 5 years, consumer-grade discs are
okay. When talking archival storage, 5 years is miniscule compared to
40 years or 100 years. If you want long-term storage (i.e., archiving),
you pay extra for archival-grade [re]writable media. Plus, you don't
want to write at the maximum density. Since it archival storage, you
don't need and probably don't want rewritable media, just writable
media. Because it is still using chemical phase change to emulate pits,
you need to properly store that archival media.

Not at the microscopic size involved, surely (either smell or warping) -
you'd only need to create a tiny black spot inside the plastic.

A file is not one or few black specks.

I think _part_ of the decay mechanism for R discs isn't just the "burnt"
bits reverting to non-burnt (if that does indeed happen), but also the
non-burnt parts may darken with time (especially if not kept out of the
light), thus reducing the difference between the two, which is after all
what's important.

The chemical phase change is not permanent, plus heat and sunlight can
affect it, too.

https://en.wikipedia.org/wiki/GeSbTe

For archival storage, the concern is data rot over time. A chemical
phase change doesn't last forever. My recollection is consumer-grade
discs have about a 5-year before data rot starts. The far better
quality but more expensive archival discs are rated to 100 years before
data rot starts (assuming proper storage).

https://www.verbatim.com/subcat/opti...de-gold-dvd-r/

In message , VanguardLH
writes:
"J. P. Gilliver (John)" wrote:

Ah. $39.95 [for IsoBuster].

Use the free version. During installation, opt for the free version.
Yep, that means some features are disabled.

https://www.isobuster.com/license-models.php

Ah. When I looked at that page in my default browser, I only saw the
left-column menu - the text and comparison table were completely
invisible! Downloaded. [Hey, it can't be any good - it's only 4 MB (-:!]

There are certainly alternatives to IsoBuster, but that's the one that I
know about. I haven't installed it in my latest computer build, but
that would be what I'd use if I needed its features.

I haven't installed it yet. If I do, I hope it takes over the necessary
parts of the OS, so that when I put a corrupted disc in, the system
doesn't almost seize up. (And yes, I do have autorun etc. turned off.]

Seems odd that the industry didn't make a version that defaulted to
letting you access existing copies of unchanged files; well, I suppose
not so odd in that they want to sell more and more blanks, but as the
drive manufacturers weren't always the same companies as the blank
manufacturers, you'd have thought one would have broken ranks ...

You can certainly NOT close a session. You could then keep "burning"
more files to the same session. Alternatively, you can create a multi-
session disc: if you copy the same file (changed or not), the old file
remains but the TOC will point to the last saved file by the same name
and path.

https://www.lifewire.com/how-to-crea...n-disc-2438838

Reading the IsoBuster page, I remembered that the giant-floppy format I
used to read was called UDF. As IsoBuster called it, it was
drag-and-drop. I only used it on RW discs; I do remember you could use
it on R discs, but I didn't think the extra convenience was worth the
reduced capacity.

The OP was interested in long term storage which makes it appear the OP
wants archival storage, and you close those sessions to protect the
files against accidental deletion. You're not supposed to keep
modifying archival media. That's why I said backups are not the same as
archiving.

Agreed. Which is why I was a bit puzzled when we were talking about RW
media (not sure if you or the OP started that, you I think); I always
thought RW were (by nature of their RWness) less reliable than R.
[]
Not at the microscopic size involved, surely (either smell or warping) -
you'd only need to create a tiny black spot inside the plastic.

A file is not one or few black specks.

Per bit, I meant, obviously! (And I'm aware of the complex coding
involved too.)
[]
--
J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf

The trouble with the death penalty has always been that nobody wanted it for
everybody, but everybody differed about who should get off. - Albert
Pierrepoint, in his 1974 autobiography.