If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
Long Term Data Storage
I need the best long term data storage available.
Recommendations please. Forget about CD or DVD since I need read/write easily to one media. I have about 500 GBytes up to 1 TByte to maintain. Speed is not important. Thanks ! |
Ads |
#2
|
|||
|
|||
Long Term Data Storage
On 01/02/2020 19:38, zernot wrote:
I need the best long term data storage available. Recommendations please. Forget about CD or DVD since I need read/write easily to one media. I have about 500 GBytes up to 1 TByte to maintain. Speed is not important. Thanks ! External hard drive or Google Cloud Storage or Microsoft Azure or Amazon Web Services or Alibaba or IBM Cloud or anything that turns you on. What's your budget like for this task? Nym-Shifter has struck again!!!!!!!!!!!!!!!!! Path: aioe.org!news.mixmin.net!news.unit0.net!news.netfr ont.net!.POSTED.107.184.178.24!not-for-mail From: zernot Newsgroups: microsoft.public.windowsxp.general,alt.windows7.ge neral Subject: Long Term Data Storage Date: Sat, 1 Feb 2020 11:38:32 -0800 Organization: Netfront http://www.netfront.net/ Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Date: Sat, 1 Feb 2020 19:48:47 -0000 (UTC) Injection-Info: adenine.netfront.net; posting-host="107.184.178.24"; logging-data="16868"; " User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0 SeaMonkey/2.46 X-Mozilla-News-Host: news://freenews.netfront.net:119 Xref: aioe.org microsoft.public.windowsxp.general:155579 alt.windows7.general:48359 -- With over 1.2 billion devices now running Windows 10, customer satisfaction is higher than any previous version of windows. |
#3
|
|||
|
|||
Long Term Data Storage
zernot wrote:
I need the best long term data storage available. Recommendations please. Forget about CD or DVD since I need read/write easily to one media. I have about 500 GBytes up to 1 TByte to maintain. Speed is not important. One machine, no redundancy, another sata hdd. Network, no redundancy rig a RPi4 plus USB3 sata hdd NAS. Network redundancy, rpi4 + 2 usb3 satas. -- Mike Easter |
#4
|
|||
|
|||
Long Term Data Storage
Is there code available for the RPi4 to do the LAN to USB drives or is that standard internal code ? Links please. Mike Easter wrote: zernot wrote: I need the best long term data storage available. Recommendations please. Forget about CD or DVD since I need read/write easily to one media. I have about 500 GBytes up to 1 TByte to maintain. Speed is not important. One machine, no redundancy, another sata hdd. Network, no redundancy rig a RPi4 plus USB3 sata hdd NAS. Network redundancy, rpi4 + 2 usb3 satas. |
#5
|
|||
|
|||
Long Term Data Storage
On 2020-02-01, zernot wrote:
I need the best long term data storage available. Stone tablets. Not very fast, and limited storage density, but they have been known to survive thousands of years under extreme conditions. -- ----------------------------------------------------------------------------- Roger Blake (Posts from Google Groups killfiled due to excess spam.) The US Census vs. privacy -- http://censusfacts.info Don't talk to cops! -- http://www.DontTalkToCops.com Badges don't grant extra rights -- http://www.CopBlock.org ----------------------------------------------------------------------------- |
#6
|
|||
|
|||
Long Term Data Storage
zernot wrote:
Is there code available for the RPi4 to do the LAN to USB drives or is that standard internal code ? There are lots of articles about doing it and doing it different ways all the way from people creating enclosures for the rpi and the drives using 3D printer to using commercially available hat for the rpi for multiple sata drives. I'll pick an example of an easy one that doesn't mention the rpi4 which I think would be the easiest because it already has the usb3 if that is what you do for the hdd. Since this is in a Win group, I'll pick one which (also) uses Win putty/ssh for the headless operation of the rpi. The article also has links for doing it w/ linux as well. https://howtoraspberrypi.com/create-...-pi-and-samba/ Create a NAS with your Raspberry Pi and Samba That article assumes you are ssh connected, which is described he https://howtoraspberrypi.com/ssh-ras...emote-control/ Learn how to use SSH to remote control your Raspberry Pi The first link above uses Samba for the NAS: "Creating the NAS server with Samba" -- Mike Easter |
#7
|
|||
|
|||
Long Term Data Storage
Thanks Mike.
Ordered RPi 4B kit and receiving it tomorrow. This will be a Windows to LAN to RPi to USB. Limitations ? Where else to go for info. i.e. newsgrops etc. Thanks again. Mike Easter wrote: zernot wrote: Is there code available for the RPi4 to do the LAN to USB drives or is that standard internal code ? There are lots of articles about doing it and doing it different ways all the way from people creating enclosures for the rpi and the drives using 3D printer to using commercially available hat for the rpi for multiple sata drives. I'll pick an example of an easy one that doesn't mention the rpi4 which I think would be the easiest because it already has the usb3 if that is what you do for the hdd. Since this is in a Win group, I'll pick one which (also) uses Win putty/ssh for the headless operation of the rpi. The article also has links for doing it w/ linux as well. https://howtoraspberrypi.com/create-...-pi-and-samba/ Create a NAS with your Raspberry Pi and Samba That article assumes you are ssh connected, which is described he https://howtoraspberrypi.com/ssh-ras...emote-control/ Learn how to use SSH to remote control your Raspberry Pi The first link above uses Samba for the NAS: "Creating the NAS server with Samba" |
#8
|
|||
|
|||
Long Term Data Storage
zernot wrote:
Thanks Mike. YW. Ordered RPi 4B kit and receiving it tomorrow. This will be a Windows to LAN to RPi to USB. Limitations ? Well; it depends on how easily you can adapt to the linux system 'engaged' w/ Win. Where else to go for info.Â* i.e. newsgrops etc. There's a rpi group which has some pretty knowledgeable folks, some are very hardware oriented. For this kind of project, there's a lot of articles online you can find w/ a search. Some people have made some 'serious' NAS w/ RPi base, others have simpler systems. The rpi group comp.sys.raspberry-pi My own experience w/ RPi is just a 'simple' RPi3B which has its own monitor kb mouse, not NAS and I'm familiar w/ using llnux systems but not remotely, but networked w/ Win systems in the network. And I read about stuff like this, as opposed to 'doing'. -- Mike Easter |
#9
|
|||
|
|||
Long Term Data Storage
zernot wrote:
Is there code available for the RPi4 to do the LAN to USB drives or is that standard internal code ? Links please. If you're asking questions like this, an "appliance" is probably a better answer. RPi is a bit deceptive, in that the base computer board is cheap, but most people will want a housing, power supply, and other do-dads to go with it. To make a complete product. It would end up almost as expensive as one of these. Synology DS120j BYOD (Bring your own disk) NAS (Network Attached Storage) $100 https://www.newegg.com/synology-ds12...82E16822108637 One 3.5" drive bay Two USB3 connectors for external devices That's about as cheap as they get today. A two bay unit, needs a bit more electricity, but I don't know if it justifies a $167 price (when it has no disks inside at purchase). Synology 2 bay NAS DiskStation DS218j (Diskless) $167 https://www.newegg.com/synology-ds21...82E16822108688 Synology Hybrid RAID (SHR), Basic, JBOD, RAID 0, RAID 1 Two 3.5" drive bays Two USB3 connectors for external devices These usually support fairly high capacity disks. You want to check the capacity figure, before you buy. ******* You can occasionally find a review. https://www.smallnetbuilder.com/nas/...tions-reviewed With two disk bays, you can place two 1TB drives and operate RAID 1. If one disk dies, the theory is the second disk continues to operate. RAID 1 mode is "Mirror" mode, so the disks are supposed to remain identical. To achieve that, you might want to power it from a UPS, and do orderly shutdowns when the power fails and the unit remains powered by the UPS. Externally, it looks like a single 1TB volume, but it's stored on two identical disks. If one disk is fried, the second disk still has your data. I find the packaging for my home projects costs a fortune, which is why I'd want to do the maths on an RPi project carefully before going that route. On the minus side, the software and interfaces that come with NAS products, aren't everyones cup of tea. So there's that. I'm a little too cheap for this sort of stuff, and you won't catch me buying stuff like this. I just plug a disk into the machine and do whatever is necessary that way. The disks on my machines load from the side. My typing machine has a "rack" of sorts next to the machine, and drives go into that thing, rather than into 5.25" bays in the main case. So when I mention putting disks in the machine, I'm sure some people think I'm struggling with 5.25" holes on the front of the machine, whereas I'm just sliding Antec trays into the side of the machine. That's a fair bit easier. And I have plenty of redundancy and way more drives than it is wise to own. I have at least two dozen drives now. The problem with that many drives, is never having a full inventory of what's on them :-) I do have an "Inventory" folder, but it's completely out of date. And what you'd be using on the NAS, is SMB/CIFS (Windows File Sharing). If you've had trouble with that, then that factors into your decision to go this route. Some people never get Windows File Sharing working worth a darn, and part of the reason is their having an excessively complicated network setup. (They don't know how their networking works, or doesn't work.) I can tell you when I first started file sharing with the OSes more modern than WinXP, there were times I wanted to throw stuff out the window. But today, it all works. Even if I have to use IP addresses to address stuff part of the time (Linux side). An alternative with the NAS, is FTP. It also has FTP protocol, suited to usage on the local LAN only. If you're having trouble getting full speed operation, and have a really big file to transfer, you can do it with FTP instead, and the transfer will go faster. I once transferred an entire hard drive using FTP :-) Paul |
#10
|
|||
|
|||
Long Term Data Storage
zernot wrote:
I need the best long term data storage available. Forget about CD or DVD since I need read/write easily to one media. I have about 500 GBytes up to 1 TByte to maintain. Speed is not important. 100GB is the largest you can go with optical media (BD 4-layer); however, writable optical media uses chemical phase change to emulate pits burned in my the commercial "printing" process, and the chemical phase change wanes over time. There are optical discs rated for archival storage of 100 years, but you have to store them out of sunlight and in a cool place. There are plans to increase optical capacity, but they'll be very pricey when they appear. For example: https://www.cdrom2go.com/blog/the-fu...d-optical-disc A problem with optical media is its capacity decreases with subsequent write sessions. When you write data to the optical disc, you're likely to close the session to prevent accidental deletion of the file(s). When you next write to the disc, you start a new session. The old session remains on the disc. That lets you traverse through old versions of the same file(s) by walking back through the sessions on the disc. However, eventually there is no more free space or not enough remaining free space in a new session for the size of a file you are trying to write to it. That means having to erase the disc (lose ALL files) and start a new session (where you hope writing the old files on the newly erased disc won't have a write failure, and you lose all those old file versions). That's assuming you're using rewritable media. If you're using write-one media, you can't erase it, the old sessions stay on the disc, and the disc's capacity keeps decreasing. Everyone weaned on computers in the last decade tends to forget or doesn't even know about tape storage. HDDs have a MTBF of 9 years. Tape is 46 years. Google uses tape for archival storage. There are large capacity cartridges and drives to handle them, but they're probably outside your budget. https://www.ironmountain.com/resourc...hive-solutions You never mention what hardware protocol you want to use for the backup media. SATA, USB, something else? And if you are going to keep around the old computer to make sure that the hardware protocol is still available after decades of archival storage (since computers change and old hardware protocols fade away). Why not use a USB-attached 1, 2, or 4 TB external HDD? Just remember that magnetic media suffers from dipole stress which means bit differentiation becomes more difficult; i.e., the bits will fade. Magnetic media needs refreshing (the data must be rewritten) every few years (figure on a 5-year refresh cycle for stored -- not used -- HDDs unless you rotate the HDDs sufficiently that the data doesn't survive that long). HDDs are mechanical devices, so they suffer from mechanical stress. That's why optical media is often used, because if the drive goes bad the media is still okay, and you get a replacement drive (if still available) to read the old media. If the mechanicals in an HDD go bad, you either lose that data (but you should have duplicates, anyway) or you pay a recovery lab to lift the data off the platters. If you're archiving to HDDs, you don't use those for backups. Backup and archiving are not the same thing. Backups are for disaster recovery (data or hardware). Archives are for data recovery. SSDs are not as cost effective as HDDs, especially for archival storage. Also, SSDs are destructive: the more you write on them, to close you approach their maximum write cycle count. They self-destruct as they are written. If you write just once, they'll last a very long time. The more you write to them, the shorter their remaining lifespan. They don't suffer the failure from mechanicals as do HDDs. Flash drives are also self-destructive writable devices, and the cheapies are not safe for archival storage. No matter what media you choose, you'll need to retain a working computer that provides you with all the old hardware protocols needed to support that media. SATA and USB are common now. Who knows what will get used a decade from now, and after that change it could be another decade when you can no longer get the hardware to support your old archival media. You need an archival computer that is used only to make sure that archival media remains usable. That archival computer gets stored, not used as a workstation for everyday use. After all, sometime in the future, you will need that archived computer to read your archive media along with the programs to use the recovered data. You aren't planning to have just one copy for archival storage, right? And those copies won't be stored at the same location, right? And you won't be using them for backups, right? You're just storing data files onto the archival media, right? You don't need to keep saving duplicate copies of your programs into your archive. You only need 1 to 3 copies of your installation media depending on how many places you are going to store your archives (should be more than 1, like at work and home and somewhere else to protect against fire, flood, tornado, etc). https://blog.macrium.com/what-type-o...g-e3ef2eb75152 Without a budget, no one knows what to suggest to you, plus your duration for archival is unknown. |
#11
|
|||
|
|||
Long Term Data Storage
In message , VanguardLH
writes: zernot wrote: I need the best long term data storage available. Forget about CD or Punched plastic tape perhaps - but you'd need the square miles of building to store them in. DVD since I need read/write easily to one media. I have about 500 ("Read/write easily" and "archive storage" are probably in the end conflicting rquirements.) GBytes up to 1 TByte to maintain. Speed is not important. (You say that, but I presume within limits; consider my punched tape - even with the 60-70 MPH that I think they got up to with Colossus, I suspect 500G-1T would still take a while ...) 100GB is the largest you can go with optical media (BD 4-layer); however, writable optical media uses chemical phase change to emulate pits burned in my the commercial "printing" process, and the chemical phase change wanes over time. There are optical discs rated for archival storage of 100 years, but you have to store them out of sunlight and in a cool place. There are plans to increase optical capacity, but they'll be very pricey when they appear. For example: https://www.cdrom2go.com/blog/the-fu...he-terabyte-si zed-optical-disc A problem with optical media is its capacity decreases with subsequent It's capacity for _additional_ data, yes. write sessions. When you write data to the optical disc, you're likely to close the session to prevent accidental deletion of the file(s). When you next write to the disc, you start a new session. The old session remains on the disc. That lets you traverse through old versions of the same file(s) by walking back through the sessions on the disc. However, eventually there is no more free space or not enough remaining free space in a new session for the size of a file you are trying to write to it. That means having to erase the disc (lose ALL files) and start a new session (where you hope writing the old files on the newly erased disc won't have a write failure, and you lose all those old file versions). That's assuming you're using rewritable media. If you're using write-one media, you can't erase it, the old sessions stay on the disc, and the disc's capacity for _new_ data keeps decreasing. As an aside: you said "writable optical media uses chemical phase change .... and the chemical phase change wanes over time"; I think you were describing _re_-writable media. I think write-on[c]e media really does "burn", hence the expression; any thoughts on the decay of that? (From my experience, the _re_writable stuff's reliability lifetime is _very_ short, but that may be because I used what I think was sometimes called packet format, making the CD-RW into a giant floppy [gave IIRR a reduced capacity, I think about 500M on a 700M disc, but you could erase files and the space became available again].) For -R discs, I've some that are still usable after over a decade, but some that are certainly not. (And optical disc formats - both CD and DVD - seem to have the unusual property that they really screw up the OS; for example, I have a couple of discs from a standalone DVD [video] recorder, which were never "closed" before the machine died; trying to read them on a PC not only results in a a lack of success, as you might expect, but really screws up the OS, such that to get control back, I at the very least have to eject the discs. Similarly with the CD-Rs that have failed: trying to read them makes the OS behave - hard to pin down, best description is as if it has a bad case of 'flu!) Everyone weaned on computers in the last decade tends to forget or doesn't even know about tape storage. HDDs have a MTBF of 9 years. Tape is 46 years. Google uses tape for archival storage. There are large capacity cartridges and drives to handle them, but they're probably outside your budget. https://www.ironmountain.com/resourc...best-long-term -data-archive-solutions Tape is sequential, and slow - and the drives _can_ be complex mechanically; I'd believe your claim of 46 years (though of course it must have been derived by extrapolation) for the tapes, but not the drives - complex loading mechanisms, perishing belts ... (Nevertheless, I have lived through several periods when tape was by far the cheapest - and I could believe the best volume-wise - medium for bulk storage, and if you tell me we're in such a period again, I'm happy to believe you.) You never mention what hardware protocol you want to use for the backup media. SATA, USB, something else? And if you are going to keep around the old computer to make sure that the hardware protocol is still available after decades of archival storage (since computers change and old hardware protocols fade away). Why not use a USB-attached 1, 2, or 4 TB external HDD? Just remember that magnetic media suffers from dipole stress which means bit differentiation becomes more difficult; i.e., the bits will fade. Magnetic media needs refreshing (the data must be rewritten) every few years (figure on a 5-year refresh cycle for stored -- not used -- HDDs Probably simplest to just do a copy from one drive to another, but do you know of any software that will do a refresh on a single drive - i. e. read and definitely write it back [not just not doing the write as it "knows" it hasn't changed]? (Thinking about it, probably having two equal-size partitions and just copying the structure back and forth between them would do it.) unless you rotate the HDDs sufficiently that the data doesn't survive that long). HDDs are mechanical devices, so they suffer from mechanical stress. That's why optical media is often used, because if the drive goes bad the media is still okay, and you get a replacement drive (if still available) to read the old media. If the mechanicals in an HDD go Or your tape drive, of course. Though I suspect the drive availability might be worse. bad, you either lose that data (but you should have duplicates, anyway) or you pay a recovery lab to lift the data off the platters. If you're archiving to HDDs, you don't use those for backups. Backup and archiving are not the same thing. Backups are for disaster recovery (data or hardware). Archives are for data recovery. SSDs are not as cost effective as HDDs, especially for archival storage. Also, SSDs are destructive: the more you write on them, to close you ["the closer" (-:] approach their maximum write cycle count. They self-destruct as they are written. If you write just once, they'll last a very long time. Interesting thought. Yes, probably a pretty good archival medium - though very expensive at the moment. Though have they been around long enough for true information on their _static_ bit-rot to have been obtained? (Especially the two or three bits per cell - multilevel? - types, that Paul's always being depressed about the march of.) [] No matter what media you choose, you'll need to retain a working computer that provides you with all the old hardware protocols needed to And transitional hardware - how are you going to get the data out! If I had an old PC, with working 5¼" floppy drives, and some floppies that hadn't faded, still how would I get the data out: probably through its serial port - if I could find a modern machine with a serial port! Maybe that's what you mean by: support that media. SATA and USB are common now. Who knows what will get used a decade from now, and after that change it could be another decade when you can no longer get the hardware to support your old archival media. You need an archival computer that is used only to make sure that archival media remains usable. That archival computer gets stored, not used as a workstation for everyday use. After all, sometime in the future, you will need that archived computer to read your archive media along with the programs to use the recovered data. So arguably, your (5-10 year?) refresh cycle shouldn't just refresh the data on the old media/hardware, it should move it onto more up-to-date media. Not as onerous as it sounds, as the capacity of storage media seems to shoot up faster than Moore's law - though still _fairly_ onerous, as the copying/moving will be limited by the speed of the _old_ hardware each time. (Even more of a problem in the analogue world, for two reasons: equipment to _play_ the old material [e. g. obscure videotape formats - or even: when did you last see a film projector, or reel-to-reel tape machine?]; and, analogue copying degrades each time. [OK, digitise. But that assumes you're happy you've got all that can be got: for example much film has been digitised in SD, HD ... but may contain information that makes it worth doing in UHD, or something not even thought of yet. And extraction techniques - things are always being invented, such as scanning film using IR or UV to detect and correct for scratches.]) You aren't planning to have just one copy for archival storage, right? And those copies won't be stored at the same location, right? And you won't be using them for backups, right? You're just storing data files onto the archival media, right? You don't need to keep saving duplicate copies of your programs into your archive. You only need 1 to 3 copies of your installation media depending on how many places you are going to store your archives (should be more than 1, like at work and home and somewhere else to protect against fire, flood, tornado, etc). Assuming there'll still be something - even a VM - onto which it _can_ be installed (-:. https://blog.macrium.com/what-type-o...-i-be-using-e3 ef2eb75152 Without a budget, no one knows what to suggest to you, plus your duration for archival is unknown. True. Some hints on both - plus the type of data, and its purpose - wouldn't go amiss. -- J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf The modern world so often thinks that the way to relax is by doing absolutely nothing, and I've never really understood that. Nigella Lawson in RT 2015/10/31-11/6 |
#12
|
|||
|
|||
Long Term Data Storage
"J. P. Gilliver (John)" wrote:
VanguardLH WROTE: It's capacity for _additional_ data, yes. Nope, for each *session*. You can save the SAME file in each session. You can save a different version of the file in each session. You can include a file in one session but not in another session. For example, you could delete files or exclude them in a /new/ session: the file is gone in the new session, perhaps to sync with the current state of the file system, so you are reducing data in the new session, but the new session still sucks up space on the optical media. You could keep creating new sessions to repeatedly store the same unchanged file in each session and eventually eat up all the space on the disc. You haven't saved anything "new". Typically only the last session is accessible. The old sessions are still on the disc, but are hidden. Being hidden does not erase them nor return storage space to the disc. Folks will complain they cannot write a 10KB file to a 4GB DVD because they don't know all those old sessions are still occupying space on the disc. ISObuster and other disc tools will show you any retained old sessions still on the disc. https://www.isobuster.com/multisessions.php for _new_ data Yep, in the /new/ session (with the older sessions being hidden). "New" just means another copy of the file whether it got changed or not from the copy of the file saved in an old session. So it could be "old" data by having the same file without change duplicated in the new session. As an aside: you said "writable optical media uses chemical phase change ... and the chemical phase change wanes over time"; I think you were describing _re_-writable media. Nope, write-once or re-writable discs both use a chemical phase change to emulate a pit. You don't have the glass plates with pits and press to make duplicates from a template to have a commercial-quality disc that has real pits. All you have are the burner drives that cause a chemical phase change using a laser. Your discs are "burned" (a laser causes a phase change in the chemicals), not pressed against a glass master. I think write-on[c]e media really does "burn", hence the expression; any thoughts on the decay of that? "Burn" just means using a laser to alter a chemical dye layer. More accurate would be "heated chemical phase change". Which do you think marketers are likely to use when selling their wares to the vast majority of boobs using computers? Think about it: if the laser in a disc drive were to actually burn the disc, you'd smell it plus there would be bubbling or warping caused by damaging the recording layer. Discs aren't hollow. Marketers aren't about advertising the truth. They advertise to sell. Depends on the market to which they sell regarding the expertise of that customer market. They can't be too technical nor accurate for computer wares. Tape is sequential, and slow - and the drives _can_ be complex mechanically; I'd believe your claim of 46 years (though of course it must have been derived by extrapolation) for the tapes, but not the drives - complex loading mechanisms, perishing belts ... It's the archival storage time of the media to which I referred. As with tapes, discs, or hard disks, you are reliant on the workings of a mechanical device continuing to work for years or decades. The media survives. You simply replace a malfunctioning drive when you use removable media for archival storage. With tapes and discs, replacing the drive is easy. With HDDs, you need to pay a lab to either move the platters to a replacement disk housing or get them to lift the data off the platter to copy onto a new hard disk. Did you forget that hard disks are also mechanical devices? Archival storage duration isn't measured by the lifespan of the mechanical drive, but by how long the media retains the data. Probably simplest to just do a copy from one drive to another, but do you know of any software that will do a refresh on a single drive - i. e. read and definitely write it back [not just not doing the write as it "knows" it hasn't changed]? (Thinking about it, probably having two equal-size partitions and just copying the structure back and forth between them would do it.) SpinRite and HDD Regenerator are just a couple examples of data recovery software that will test free space to make sure it is reliable for landing of a copy of data, test the used sectors to make the read will be reliable, and then copy from the old sector to the new one. The write with move performs a refresh. Alternatively, if there isn't enough free space for the entire file, memory could be used for holding a sector of a file, and then test and refresh the old sector by writing the file back to it. I used to have HD Sentinel and, as I recall, its payware versions had disk diagnostics that included data refresh. I've heard of but never used DiskFresh. Copying from one drive to another require another drive. What if all the onboard controller slots are used, so you cannot add another internal drive. Yep, you could use a USB-attached drive, but remember the data was wanted on the internal drive, so you wouldn't be copying once but twice: once from the internal drive to the external one, and again to copy from the external drive back to the internal drive. The problem with just a copy is that you assume the formatting results in a reliable copy. The tools used for refresh ensure the target to where the copy gets put is tested beforehand, and are more thorough than running "chkdsk /r" that supposedly tests the sectors before using them. I never had to refresh an HDD because those didn't last long enough in my setup to need a refresh to ensure data integrity. Typically the HDDs get replaced in about 5 years to get bigger (larger capacity) and faster ones (the drive was faster or move to a faster hardware protocol). I have had to refresh floppies where dipole stress is a worse problem. The data that wasn't readable not only became readable but it got refreshed to realign the dipoles for maximum signal strength to improve differentiation of bits on the media. Anyone who archives files but never refreshes the magnetic media is asking for corrupted or unreadable files sometime in the future when they desparately need those files. Mortgage companies are required to keep records for 40 years. No matter what media you choose, you'll need to retain a working computer that provides you with all the old hardware protocols needed to And transitional hardware - how are you going to get the data out! If I had an old PC, with working 5¼" floppy drives, and some floppies that hadn't faded, still how would I get the data out: probably through its serial port - if I could find a modern machine with a serial port! Maybe that's what you mean by: By using that archived computer that had the 5.25" and 3.5" floppy drives you used to create the archived floppies. You need to reuse those same drives when you restore from the archived floppies due to alignment problems. Never ran into the problem where you wrote a floppy disk on one computer but couldn't read it in a floppy drive on another computer? That was due to misalignment of one of the drives. Eventually they went to laser markings on the floppy (actually burned into the media; "LS" means laser servo) to eliminate the alignment problems (https://en.wikipedia.org/wiki/SuperDisk). Remember the old Iomega Zip drives that use the "floptical" Zip disks? Those were LS drives. However, by then the floppy format was undersized and too slow compared to optical media, and even Iomega Zip media faded away (I have one remaining computer with a Zip drive along with a couple storage cases full of Zip disk, but everything on them has been copied elsewhere, so they're awaiting the discard of the old computer with the Zip drive). So arguably, your (5-10 year?) refresh cycle shouldn't just refresh the data on the old media/hardware, it should move it onto more up-to-date media. That's an expensive proposition. Why buy new discs, tapes, of HDDs when the old media is perfectly still usable? The reason the refresh software is still a viable and cost effective solution to insurance of magnetic archival of data is that the hardware gets reused, not replaced. |
#13
|
|||
|
|||
Long Term Data Storage
In message , VanguardLH
writes: "J. P. Gilliver (John)" wrote: VanguardLH WROTE: It's capacity for _additional_ data, yes. Nope, for each *session*. You can save the SAME file in each session. You can save a different version of the file in each session. You can include a file in one session but not in another session. For example, you could delete files or exclude them in a /new/ session: the file is gone in the new session, perhaps to sync with the current state of the file system, so you are reducing data in the new session, but the new session still sucks up space on the optical media. You could keep creating new sessions to repeatedly store the same unchanged file in each session and eventually eat up all the space on the disc. You haven't saved anything "new". Typically only the last session is accessible. That's the important line. Yes, if you're just using remaining space for the entire new session, including [new copies of] files that haven't changed, of course it will be as you say. And yes, I know that's the default way these discs (well, the software that uses them) work. (Though there was what I think was called "packet" for RW media, that _did_ make the disc work like a giant floppy, in other words space from a deleted file _did_ become immediately available - but you had to have the "packet driver" installed on all machines. IIRR, the two main makers as the time - Roxio/adaptec and Nero, I think - had their own systems [I think adaptec's was called DirectCD, and Nero's InCD]; I am not sure they were compatible. For some reason they gave a reduction in capacity, to IIRR 5xx MB on a 700 MB CDRW. I used to use them for transferring things between computers, much as I might a USB stick now; I didn't find them all that reliable [had to reformat them after not very long], and certainly wouldn't archive to them, or to RW using even the normal burn method.) The old sessions are still on the disc, but are hidden. Being hidden does not erase them nor return storage space to the disc. Folks will complain they cannot write a 10KB file to a 4GB DVD because they don't know all those old sessions are still occupying space on the disc. ISObuster and other disc tools will show you any retained old sessions still on the disc. https://www.isobuster.com/multisessions.php Ah. $39.95 - quite reasonable, but I don't currently have a need for it. (I have vague memories of having used something freeware in the past that could see old sessions, but I'm not sure about that.) for _new_ data Yep, in the /new/ session (with the older sessions being hidden). "New" just means another copy of the file whether it got changed or not from the copy of the file saved in an old session. So it could be "old" data by having the same file without change duplicated in the new session. Seems odd that the industry didn't make a version that defaulted to letting you access existing copies of unchanged files; well, I suppose not so odd in that they want to sell more and more blanks, but as the drive manufacturers weren't always the same companies as the blank manufacturers, you'd have thought one would have broken ranks ... As an aside: you said "writable optical media uses chemical phase change ... and the chemical phase change wanes over time"; I think you were describing _re_-writable media. Nope, write-once or re-writable discs both use a chemical phase change to emulate a pit. You don't have the glass plates with pits and press to make duplicates from a template to have a commercial-quality disc that has real pits. All you have are the burner drives that cause a chemical phase change using a laser. Your discs are "burned" (a laser causes a phase change in the chemicals), not pressed against a glass master. I realise they don't have physical pits, but all that matters is that the beam either reflects or doesn't; in a pressed CD it's because the pits are a quarter wavelength (or appropriate multiple thereof) deep and reflections from them cancel reflections from the surface beside the pits or doesn't where there's no pit, in R/RW it's because the material is opaque or isn't. I think write-on[c]e media really does "burn", hence the expression; any thoughts on the decay of that? "Burn" just means using a laser to alter a chemical dye layer. More accurate would be "heated chemical phase change". Which do you think marketers are likely to use when selling their wares to the vast majority of boobs using computers? Think about it: if the laser in a disc drive were to actually burn the disc, you'd smell it plus there would be bubbling or warping caused by damaging the recording layer. Discs aren't hollow. Not at the microscopic size involved, surely (either smell or warping) - you'd only need to create a tiny black spot inside the plastic. (As is done to make the artwork in various awards - the sort given at awards ceremonies - which are a block of clear plastic with shapes inside.) Marketers aren't about advertising the truth. They advertise to sell. Depends on the market to which they sell regarding the expertise of that customer market. They can't be too technical nor accurate for computer wares. However, I'm willing to believe you're right, and it's just a "chemical phase change" for both RW and R. (Though it could be argued that blackening by burning is just a chemical change of course!) I think _part_ of the decay mechanism for R discs isn't just the "burnt" bits reverting to non-burnt (if that does indeed happen), but also the non-burnt parts may darken with time (especially if not kept out of the light), thus reducing the difference between the two, which is after all what's important. Tape is sequential, and slow - and the drives _can_ be complex mechanically; I'd believe your claim of 46 years (though of course it must have been derived by extrapolation) for the tapes, but not the drives - complex loading mechanisms, perishing belts ... It's the archival storage time of the media to which I referred. As with tapes, discs, or hard disks, you are reliant on the workings of a mechanical device continuing to work for years or decades. The media survives. You simply replace a malfunctioning drive when you use removable media for archival storage. With tapes and discs, replacing the drive is easy. With HDDs, you need to pay a lab to either move the platters to a replacement disk housing or get them to lift the data off the platter to copy onto a new hard disk. Did you forget that hard disks are also mechanical devices? Archival storage duration isn't measured by the lifespan of the mechanical drive, but by how long the media retains the data. In academic theory, yes; for practical use, there's no point in having a tape that you are absolutely sure still has your wanted bits on it, if you don't have a reader that still works (and can't get one)! I haven't looked into data tape drives for ages (probably decades), but I assume that, like other media, they've developed over the years to handle new cartridge designs to handle higher capacities: _are_ they backwards-compatible, in other words if you buy a new tape drive today, will it still be able to read a cartridge from 20 or 30 years ago, as well as today's? (It's possible they are - after all, a DVD drive can _usually_ still read a CD [not sure about a blu-ray drive] - but I don't know, as tape is a more specialised area.) Probably simplest to just do a copy from one drive to another, but do you know of any software that will do a refresh on a single drive - i. e. read and definitely write it back [not just not doing the write as it "knows" it hasn't changed]? (Thinking about it, probably having two equal-size partitions and just copying the structure back and forth between them would do it.) SpinRite and HDD Regenerator are just a couple examples of data recovery software that will test free space to make sure it is reliable for landing of a copy of data, test the used sectors to make the read will be reliable, and then copy from the old sector to the new one. The write with move performs a refresh. Alternatively, if there isn't enough free space for the entire file, memory could be used for holding a sector of a file, and then test and refresh the old sector by writing the file back to it. I used to have HD Sentinel and, as I recall, its payware versions had disk diagnostics that included data refresh. I've heard of but never used DiskFresh. I wrote the above paragraph thinking that just reading doesn't refresh, you need to rewrite, but even as I was writing it I realised that there's no strong reason that it has to be rewritten _in the same place on the disc_, so as long as it is actually written not just read, the refresh process has happened. (Well, write _with verify_ of course.) [] reliable copy. The tools used for refresh ensure the target to where the copy gets put is tested beforehand, and are more thorough than running "chkdsk /r" that supposedly tests the sectors before using them. I've often seen people disparaging chkdsk /r and other similar; I'm not saying anyone is wrong to disparage it, but I've also yet to see any actual _proof_ that it actually lies, rather than just misleads. (Though the in-drive-electronics swapping etc. may enter into that discussion.) [] Anyone who archives files but never refreshes the magnetic media is asking for corrupted or unreadable files sometime in the future when Agreed. they desparately need those files. Mortgage companies are required to keep records for 40 years. [In your country. I don't know for here (UK) - might be more, or less.] No matter what media you choose, you'll need to retain a working computer that provides you with all the old hardware protocols needed to And transitional hardware - how are you going to get the data out! If I had an old PC, with working 5¼" floppy drives, and some floppies that hadn't faded, still how would I get the data out: probably through its serial port - if I could find a modern machine with a serial port! Maybe that's what you mean by: By using that archived computer that had the 5.25" and 3.5" floppy drives you used to create the archived floppies. You need to reuse those same drives when you restore from the archived floppies due to alignment problems. Never ran into the problem where you wrote a floppy Ah, we're talking at cross-purposes here. (I agree with you about alignment issues.) [] So arguably, your (5-10 year?) refresh cycle shouldn't just refresh the data on the old media/hardware, it should move it onto more up-to-date media. That's an expensive proposition. Why buy new discs, tapes, of HDDs when the old media is perfectly still usable? The reason the refresh software is still a viable and cost effective solution to insurance of magnetic archival of data is that the hardware gets reused, not replaced. It depends _why_ you need to access the old data. If it's to do something that can only be _done_ on the old computer, then we're into a completely different ballgame - you need to look at not only data archiving, but hardware retention/maintenance (and arguably duplication for redundancy), which is a whole new vale of tears! If you only need to access the data for say legal reasons, albeit maybe "running" it on a VM simulation of the old computer, then it doesn't matter what it's on, and arguably the more modern the better. Your mortgage companies, for example, I presume (hope!) they copied anything they had on punched cards, paper tape, or even half-inch tape, onto something more modern, some while ago. (And the expense may not be the case: maintaining _really_ old media - _and the hardware needed to access it_ [with redundant duplicates] - can be expensive, and may even not be _possible_ as those with knowledge of it retire, and even die.) -- J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf Society has the right to punish wrongdoing; it doesn't have the right to make punishment a form of entertainment. This is where things have gone wrong: humiliating other people has become both a blood sport and a narcotic. - Joe Queenan, RT 2015/6/27-7/3 |
#14
|
|||
|
|||
Long Term Data Storage
"J. P. Gilliver (John)" wrote:
Ah. $39.95 [for IsoBuster]. Use the free version. During installation, opt for the free version. Yep, that means some features are disabled. https://www.isobuster.com/license-models.php There are certainly alternatives to IsoBuster, but that's the one that I know about. I haven't installed it in my latest computer build, but that would be what I'd use if I needed its features. Seems odd that the industry didn't make a version that defaulted to letting you access existing copies of unchanged files; well, I suppose not so odd in that they want to sell more and more blanks, but as the drive manufacturers weren't always the same companies as the blank manufacturers, you'd have thought one would have broken ranks ... You can certainly NOT close a session. You could then keep "burning" more files to the same session. Alternatively, you can create a multi- session disc: if you copy the same file (changed or not), the old file remains but the TOC will point to the last saved file by the same name and path. https://www.lifewire.com/how-to-crea...n-disc-2438838 The OP was interested in long term storage which makes it appear the OP wants archival storage, and you close those sessions to protect the files against accidental deletion. You're not supposed to keep modifying archival media. That's why I said backups are not the same as archiving. I realise they don't have physical pits, but all that matters is that the beam either reflects or doesn't; It is because it is chemical for why cheap media is not applicable for archival use. Remember the OP is looking for long time storage (but didn't say how long). Yep, you want the laser reaction to continue to work, but that means you are relying on the integrity of the media. For backups that maybe have a retention of 5 years, consumer-grade discs are okay. When talking archival storage, 5 years is miniscule compared to 40 years or 100 years. If you want long-term storage (i.e., archiving), you pay extra for archival-grade [re]writable media. Plus, you don't want to write at the maximum density. Since it archival storage, you don't need and probably don't want rewritable media, just writable media. Because it is still using chemical phase change to emulate pits, you need to properly store that archival media. Not at the microscopic size involved, surely (either smell or warping) - you'd only need to create a tiny black spot inside the plastic. A file is not one or few black specks. I think _part_ of the decay mechanism for R discs isn't just the "burnt" bits reverting to non-burnt (if that does indeed happen), but also the non-burnt parts may darken with time (especially if not kept out of the light), thus reducing the difference between the two, which is after all what's important. The chemical phase change is not permanent, plus heat and sunlight can affect it, too. https://en.wikipedia.org/wiki/GeSbTe For archival storage, the concern is data rot over time. A chemical phase change doesn't last forever. My recollection is consumer-grade discs have about a 5-year before data rot starts. The far better quality but more expensive archival discs are rated to 100 years before data rot starts (assuming proper storage). https://www.verbatim.com/subcat/opti...de-gold-dvd-r/ |
#15
|
|||
|
|||
Long Term Data Storage
In message , VanguardLH
writes: "J. P. Gilliver (John)" wrote: Ah. $39.95 [for IsoBuster]. Use the free version. During installation, opt for the free version. Yep, that means some features are disabled. https://www.isobuster.com/license-models.php Ah. When I looked at that page in my default browser, I only saw the left-column menu - the text and comparison table were completely invisible! Downloaded. [Hey, it can't be any good - it's only 4 MB (-:!] There are certainly alternatives to IsoBuster, but that's the one that I know about. I haven't installed it in my latest computer build, but that would be what I'd use if I needed its features. I haven't installed it yet. If I do, I hope it takes over the necessary parts of the OS, so that when I put a corrupted disc in, the system doesn't almost seize up. (And yes, I do have autorun etc. turned off.] Seems odd that the industry didn't make a version that defaulted to letting you access existing copies of unchanged files; well, I suppose not so odd in that they want to sell more and more blanks, but as the drive manufacturers weren't always the same companies as the blank manufacturers, you'd have thought one would have broken ranks ... You can certainly NOT close a session. You could then keep "burning" more files to the same session. Alternatively, you can create a multi- session disc: if you copy the same file (changed or not), the old file remains but the TOC will point to the last saved file by the same name and path. https://www.lifewire.com/how-to-crea...n-disc-2438838 Reading the IsoBuster page, I remembered that the giant-floppy format I used to read was called UDF. As IsoBuster called it, it was drag-and-drop. I only used it on RW discs; I do remember you could use it on R discs, but I didn't think the extra convenience was worth the reduced capacity. The OP was interested in long term storage which makes it appear the OP wants archival storage, and you close those sessions to protect the files against accidental deletion. You're not supposed to keep modifying archival media. That's why I said backups are not the same as archiving. Agreed. Which is why I was a bit puzzled when we were talking about RW media (not sure if you or the OP started that, you I think); I always thought RW were (by nature of their RWness) less reliable than R. [] Not at the microscopic size involved, surely (either smell or warping) - you'd only need to create a tiny black spot inside the plastic. A file is not one or few black specks. Per bit, I meant, obviously! (And I'm aware of the complex coding involved too.) [] -- J. P. Gilliver. UMRA: 1960/1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf The trouble with the death penalty has always been that nobody wanted it for everybody, but everybody differed about who should get off. - Albert Pierrepoint, in his 1974 autobiography. |
|
Thread Tools | |
Display Modes | |
|
|