Sync apps

Peter Kozlov wrote:
On 2018-06-09, Frank Slootweg wrote:
Peter Kozlov wrote:
On 2018-06-09, Frank Slootweg wrote:
Wolf K wrote:
On 2018-06-07 16:54, Peter Kozlov wrote:
Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

Does that make sense?

Looking for experienced answers. No Google'd replies needed. I can
Google, too.

Thanks in advance.

I assume A is your working disk, and B is your archive.

Use one of the backup programs that will do incremental backups. After
the first backup, only changed or new files are backed up. So if you
backup A to B, B will have what is no longer on A plus whatever is new
or different.

Exactly! For example, I use Cobian Backup to do a one-way 'sync' by
using Cobian Backup's Incremental backup. No need for a seperate 'sync'
program.

Also the opposite is often true: A 'sync' program which can do a
one-way sync is effectively an incremental backup program. I
realized/used that [1] when I couldn't find a good incremental backup
program for backing up an Android device to a (SMB) Network Share.

[1] SyncMe Wireless
https://play.google.com/store/apps/details?id=com.bv.wifisync

What is the backup? Is it just a collection of individual files easily
accessible to anyone plugging in the B drive, or is software required
to restore and read the backup. If the latter, this is not what I am
looking for. The one way sync would be better.

The backup is just files in folders, organized the same as the source
(A). Cobian Backup *can* compress or/and encrypt the backed up files,
but by default it does not do that.

So yes, "Is it just a collection of individual files easily accessible
to anyone plugging in the B drive".

The fun part is that for this very reason, Cobian Backup only has a
backup facility, no restore facility. And the author - rightly so -
refuses to add a restore facility, despite - clueless - users asking for
one.

This does sound viable then. I'll look at it. I assume it has a
scheduling facility?

Yes, it has, with quite a lot of functionality.

However, I wouldn't switch - from whatever backup software you're
currently using - to Cobian Backup, just for this 'one-way sync'
functionality.

My response was just in support of Wolf K's comment to "Use one of the
backup programs that will do incremental backups.", not so much a
recommendation to switch - from your current backup software - to Cobian
Backup [1].

BTW, which backup software do you currently use and doesn't that also
have this 'one-way sync'/incremental backup functionality?

[1] http://www.cobian.se/cobianbackup.htm

N.B., the Cobian Backup site doesn't specifically mention Windows 10
support. But it also doesn't mention Windows 8[.1] support and it runs
fine on 8.1.

On 2018-06-09, Frank Slootweg wrote:
Peter Kozlov wrote:

[snip]

This does sound viable then. I'll look at it. I assume it has a
scheduling facility?

Yes, it has, with quite a lot of functionality.

However, I wouldn't switch - from whatever backup software you're
currently using - to Cobian Backup, just for this 'one-way sync'
functionality.

I'm using Windows' created recovery drive and restore points as the
front line to solve a problem. On top of that I am using Macrium Reflect
as suggested here for my full-covery oh **** moments.

What this is for is totally different. I have large files that come in
on a daily basis. I work on them in a shared environment for about an
hour or two. I separate them by client. These build up very quickly.
Within a month I have about 5 TBs of data. This is all on a file server.
The volume is so high that all that data cannot sit in the file server.
I push it out almost on a bi-monthly basis. I have it go to external
hard drive. And then I create a simple text file so I can search for
what I want and know where to locate it. What drive is it on for
example.

I want to simplify it so I can hand it off to someone else who's not
technical at all. Ideally, a schedule will run and do this. When A is
full, B will be full and B can be removed. A can be deleted and then the
whole process starts over again. So over time we'll have a lot of B
drives and a lot of text files with lists of files on these drives. Need
a file from the past, search the text files, locate what you want, find
the drive and pull the file you need. The files are typically between 9
GBs and 400 GBs each. They are media files is why they are so large. No
compression needed. Just basic file copies of the critical files in a
simple, accessible method.

My response was just in support of Wolf K's comment to "Use one of the
backup programs that will do incremental backups.", not so much a
recommendation to switch - from your current backup software - to Cobian
Backup [1].

I understand.

BTW, which backup software do you currently use and doesn't that also
have this 'one-way sync'/incremental backup functionality?

The server is backed up by someone else. It is of no concern to me.
The data flowing into the media area is all I am concerned with. As long
as I have a backup of each file I am good. I may even make a backup of
the backup. B might get a C backup so there are always two.

Other than that, the workstation which is a Windows 10 machine with
expensive software that has to be activated via a machine finger-print
isn't something I want to have to redo. This is where Macrium comes in.
Should that machine need a full recovery, that's what I will use. This
machine needs to be operational on a daily basis.

[1] http://www.cobian.se/cobianbackup.htm

N.B., the Cobian Backup site doesn't specifically mention Windows 10
support. But it also doesn't mention Windows 8[.1] support and it runs
fine on 8.1.

Well, I am tasting a sync app. I was just looking at it Friday and then
I had to run for an appoitment that got moved up early. So I don't
remember what it was called. I can reply again on Monday and say what it
is. It's basically a classic sync app. I have all of 1 minute on it so I
don't remember the name of it. I will test it and see if it checks off
all the boxes I'm interested in.

I'm hoping to use 2 TB external portable hard drives in exFAT as the
file store. I don't like that exFAT isn't journaled, but I do like how
exFAT is basically read/write compatible on the Mac right out of the
box. That is important to me. Most Linux installs also have a FUSE
support for exFAT installed or in the case of SUSE Tumbleweed, it's a
simple install. Of course Windows supports this. So I plug the drives
into anything and read write to them. I don't need to deal with
restoration at all. Just need basic access to files. That doesn't happen
often. Most of the time the files will just sit on the drive in storage.

--
Peter Kozlov

On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

--
Peter Kozlov

Microsoft's SyncToy does all of that. It's downloadable from Microsoft
for free. You can set it save the settings of jobs on certain sets of
folders over and over again.

Yousuf Khan

Download SyncToy 2.1 from Official Microsoft Download Center
https://www.microsoft.com/en-ca/down....aspx?id=15155

On 6/7/2018 4:54 PM, Peter Kozlov wrote:
Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

Does that make sense?

Looking for experienced answers. No Google'd replies needed. I can
Google, too.

Thanks in advance.

On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?

On Sun, 10 Jun 2018 09:07:07 -0700, Bennett
wrote:

On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?



There are three downsides:

1. It's not free.

2. If you have a lot of data, it takes a long time to restore it.

3. The risk is probably small, but there's a risk that some
unauthorized person will see your data.

Those downsides are more significant to some people than to others,
and because there are also obvious upsides (automatic backup whenever
something changes, no risk of a burglar stealing your backup, etc.)
whether or not to use such a service is up to you. Personally, I think
it's a good choice for most people.

On 2018-06-10, Bennett wrote:
On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?

The bandwidth isn't there for something like that. I'm ingesting far too
much data for that to be an option at all. And the data isn't simple one
machine. It's non-machine specific.

There is a flow of large data-sets that arrive into machine A. And
machine A processes these into a share. From the share they are used by
a few different people. After a short while, the work is completed, but
the stays put for a while longer. This is happening multiple times per
day.

At the end of the day, everything new on A that doesn't exist on B is
pushed to B. As A becomes full, data-sets from A are deleted. They can
be deleted because there are backups for long term offline on B.

When B becomes full it is removed and stored for long-term. A new blank
B drive is attached and A is completely cleared. At this point, the
process starts all over again.

The daily amount of data can be is little as 9 GBs but that is rare.
It's typically between 150 GBs and 400 GBs. Sometimes it is much more. I
think I had to deal with 3 TBs one day. So the bandwidth I have to play
with is already stretched to the limites as it is. I can't afford to
allow something like Carbonite or any web-based solution because I need
all of my bandwidth. Much of what I am describing can come in over the
internet or on physical drives. As it is sometimes yesterday's work
bleeds into today, which is bad. That can start me off with new work
being delated while old work is completed. That drives me crazy.

What I need is a lot more bandwidth. And I need faster copies. This used
to be no more 50 GBs in a day and USB2 could be used for the backups
without too much trouble. It's not at a point where I am using SSDs on
much of this because that gives me about 329 MB/sec copy speed. Gigabit
Ethernet is quickly becoming a problem as well.

This isn't like backing a computer with small delta changes per day
that an ordinary user users. This is an ongoing archive or large
data-sets that flow through a media company. The footage is getting
larger and larger and resolution gets better and better. Image a 2 hour
movie in HD quality with something like ProRes. It's can easily be 220
GBs. And now 4k is a thing and 8k will follow. Imagine archiving show
after show after show in an endless number of on-going shows. Even on
very large arrays everything can't be kept on-line all the time. Things
have to be moved off at a pace to keep the flow flowing through. I can't
remove anything without having of a backup of it. And I have to give it
to other parties.

--
Peter Kozlov

Bennett wrote:

What's the downside of using something like Carbonite or other real time
cloud based software?

How many users have "Kansas grade broadband" ?

https://fiber.google.com/cities/kansascity/

And even if the access infrastructure was sufficient
to allow everyone to do that, it costs around $0.03
per gigabyte to shift things over a trunk. If you
upload a terabyte to the cloud, it's costing your
ISP around $30 to provide that service. If everyone
started doing that, then all the monthly fees would
go up by $30 a month.

Even for Google, their costing model is for
"sporadic content consumption", not "continuous uploads".
I'm sure Google even hates Netflix usage, but won't
publicly say that of course.

Google has stopped their expansion plans, so you
cannot assume they will blanket the country
with that kind of service. I think they've done
the math. Maybe they'll buy an interest in
someones LEO satellite network some day, and
that's the next thing you'll hear about. Could
easily take another five years.

And since it's LEO, once the satellites are
airborne, they'll have to start replacing
the first ones. For a satellite system
like that, it's a "continuous launch" model.
GEO is what regular satellite Internet uses
(large satellites, geosynchronous orbit, bird
flies for as long as it has fuel and power).
The LEO satellites will be smaller and gutless.
And fall from orbit a lot easier. The pricing then,
should have a lot to do with launch costs.

Paul

Ken Blake wrote:

Bennett wrote:

What's the downside of using something like Carbonite or other real
time cloud based software?

There are three downsides:

1. It's not free.

If your upload size isn't super high, you can use OneDrive or Google
Drive to store the backup files (so don't create backup images of
partitions but just data-only files in the backup). You store the
backup in a local folder (preferrably NOT in the same partition as what
gets backed up and also preferrably not in a partition on the same drive
as the data files getting backed up). After saving the backup there,
the local client for OneDrive or Google Drive will upload a copy
although the upload is very slow (it is throttled and will be even
slower than whatever is your upstream bandwidth).

2. If you have a lot of data, it takes a long time to restore it.

And even far longer to upload the data in the first place. Most users
have asymmetrical Internet service: fast downstream bandwidth but very
slow (by comparison) upstream bandwidth. Only use for data-only uploads
and then calculate how long all those bytes will take to upload. Could
be hours or even days depending on the upload size.

3. The risk is probably small, but there's a risk that some
unauthorized person will see your data.

Most backup programs let you password secure the backup file. If you
just copy files, like what the OP is doing, yep, you expose your
personal information on the server should your online account get
hacked. Make sure to use a STRONG password and optionally employ any
additional security mechanisms the site offers. If you zip up the files
(since you're likely to have an unzip available anywhere for free), you
can password-protect the .zip archive file; however, as I recall, it is
not a high-secure mechanism but then you have your account login along
with a .zip password, so more hurdles for a hacker to overcome.

Those downsides are more significant to some people than to others,
and because there are also obvious upsides (automatic backup whenever
something changes, no risk of a burglar stealing your backup, etc.)
whether or not to use such a service is up to you. Personally, I think
it's a good choice for most people.

https://www.carbonite.com/blog/artic...-reuse-attack/

Use a unique password at every domain. Do not reuse your login
credentials amongst multiple sites. If one gets hacked, and if there is
any info leading to your other accounts or the hackers just attack the
common sites, they have your login credentials to use elsewhere if you
use the same login. There is software to assist in creating unique
passwords everywhere. I use an algorithm that I can remember in my head
that gives a strong and unique password at every site.

On Sun, 10 Jun 2018 16:53:09 -0000 (UTC), Peter Kozlov
wrote:

On 2018-06-10, Bennett wrote:
On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?

The bandwidth isn't there for something like that. I'm ingesting far too
much data for that to be an option at all. And the data isn't simple one
machine. It's non-machine specific.

There is a flow of large data-sets that arrive into machine A. And
machine A processes these into a share. From the share they are used by
a few different people. After a short while, the work is completed, but
the stays put for a while longer. This is happening multiple times per
day.

At the end of the day, everything new on A that doesn't exist on B is
pushed to B. As A becomes full, data-sets from A are deleted. They can
be deleted because there are backups for long term offline on B.

When B becomes full it is removed and stored for long-term. A new blank
B drive is attached and A is completely cleared. At this point, the
process starts all over again.

The daily amount of data can be is little as 9 GBs but that is rare.
It's typically between 150 GBs and 400 GBs. Sometimes it is much more. I
think I had to deal with 3 TBs one day. So the bandwidth I have to play
with is already stretched to the limites as it is. I can't afford to
allow something like Carbonite or any web-based solution because I need
all of my bandwidth. Much of what I am describing can come in over the
internet or on physical drives. As it is sometimes yesterday's work
bleeds into today, which is bad. That can start me off with new work
being delated while old work is completed. That drives me crazy.

What I need is a lot more bandwidth. And I need faster copies. This used
to be no more 50 GBs in a day and USB2 could be used for the backups
without too much trouble. It's not at a point where I am using SSDs on
much of this because that gives me about 329 MB/sec copy speed. Gigabit
Ethernet is quickly becoming a problem as well.

This isn't like backing a computer with small delta changes per day
that an ordinary user users. This is an ongoing archive or large
data-sets that flow through a media company. The footage is getting
larger and larger and resolution gets better and better. Image a 2 hour
movie in HD quality with something like ProRes. It's can easily be 220
GBs. And now 4k is a thing and 8k will follow. Imagine archiving show
after show after show in an endless number of on-going shows. Even on
very large arrays everything can't be kept on-line all the time. Things
have to be moved off at a pace to keep the flow flowing through. I can't
remove anything without having of a backup of it. And I have to give it
to other parties.

Your described use case is Machine A backs up to Machine B every day,
and when Machine B is full you swap it for a new (empty) Machine B,
right?

I think I would skip the second part of that and simply use Machine B as
a FIFO buffer, so rather than wait for B to be full and then swap it for
a new B, I would pull data off of B every evening, or as often as it
takes, so that as far as A is concerned, B is never full.

Regarding network connections, you have USB3, USB3.1, and of course
10Gig Ethernet if you're serious about speed. 10G NICs are fairly
affordable now, starting at about $30. But back to the paragraph above,
if B is a simple FIFO buffer, then you can constantly pull data off of B
while A is putting more data on. As far as A is concerned, B's capacity
is infinite.

Char Jackson wrote:
On Sun, 10 Jun 2018 16:53:09 -0000 (UTC), Peter Kozlov
wrote:

On 2018-06-10, Bennett wrote:
On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.
What you are describing is an incremental backup. You can do this with
Macrium Reflect.
I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?
I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?
The bandwidth isn't there for something like that. I'm ingesting far too
much data for that to be an option at all. And the data isn't simple one
machine. It's non-machine specific.

There is a flow of large data-sets that arrive into machine A. And
machine A processes these into a share. From the share they are used by
a few different people. After a short while, the work is completed, but
the stays put for a while longer. This is happening multiple times per
day.

At the end of the day, everything new on A that doesn't exist on B is
pushed to B. As A becomes full, data-sets from A are deleted. They can
be deleted because there are backups for long term offline on B.

When B becomes full it is removed and stored for long-term. A new blank
B drive is attached and A is completely cleared. At this point, the
process starts all over again.

The daily amount of data can be is little as 9 GBs but that is rare.
It's typically between 150 GBs and 400 GBs. Sometimes it is much more. I
think I had to deal with 3 TBs one day. So the bandwidth I have to play
with is already stretched to the limites as it is. I can't afford to
allow something like Carbonite or any web-based solution because I need
all of my bandwidth. Much of what I am describing can come in over the
internet or on physical drives. As it is sometimes yesterday's work
bleeds into today, which is bad. That can start me off with new work
being delated while old work is completed. That drives me crazy.

What I need is a lot more bandwidth. And I need faster copies. This used
to be no more 50 GBs in a day and USB2 could be used for the backups
without too much trouble. It's not at a point where I am using SSDs on
much of this because that gives me about 329 MB/sec copy speed. Gigabit
Ethernet is quickly becoming a problem as well.

This isn't like backing a computer with small delta changes per day
that an ordinary user users. This is an ongoing archive or large
data-sets that flow through a media company. The footage is getting
larger and larger and resolution gets better and better. Image a 2 hour
movie in HD quality with something like ProRes. It's can easily be 220
GBs. And now 4k is a thing and 8k will follow. Imagine archiving show
after show after show in an endless number of on-going shows. Even on
very large arrays everything can't be kept on-line all the time. Things
have to be moved off at a pace to keep the flow flowing through. I can't
remove anything without having of a backup of it. And I have to give it
to other parties.

Your described use case is Machine A backs up to Machine B every day,
and when Machine B is full you swap it for a new (empty) Machine B,
right?

I think I would skip the second part of that and simply use Machine B as
a FIFO buffer, so rather than wait for B to be full and then swap it for
a new B, I would pull data off of B every evening, or as often as it
takes, so that as far as A is concerned, B is never full.

Regarding network connections, you have USB3, USB3.1, and of course
10Gig Ethernet if you're serious about speed. 10G NICs are fairly
affordable now, starting at about $30. But back to the paragraph above,
if B is a simple FIFO buffer, then you can constantly pull data off of B
while A is putting more data on. As far as A is concerned, B's capacity
is infinite.

At that rate, after maybe ten years, the archive could be
the size of the initial archive.org .

Something has to pay for that storage.

Queuing it up, planning enough "sinks", that's easy.

Take the number of hours in the day, the amount of data,
and that tells you the megabytes per hour performance level
you need. Whether it's LTO or optical discs, all the devices
will have a spec for write rate. Plan for N of them, to
sink at the required rate. If your queue array is big enough,
you'll be able to handle the "average" sink rate, with some
room for days when above average input happens.

For redundancy, you can use something like PAR or equivalent.
So that you write N+1 pieces of media, and if a piece of
media is a complete failure (laser cannot track groove in
disc), you have a solution.

But at the end of the day, you need to know what an
LTO tape costs, what a reliable BluRay write-once disc
costs, to figure out which to pay for. Using hard drives
would likely be more expensive.

If restoration is a real possibility, you also need to consider
the restoration rate.

To give an example, we had a network backup system at work
(backups were offsite). There was a failure at our site,
caused by electrical work by the hydro company, which damaged
a ton of computers. It took *two weeks* for me to get my hard
drive restored. I was on a loaner machine for those two weeks.
My home directory (fortunately), wasn't stored on the machine
local drive. But in terms of restore speed, the platform
used was obviously swamped. *Everything* at work was
backed up, but without a service guarantee on restoration.
If there hadn't been a hydro event, I'm sure I could have
been restored in only a couple hours when the facilities
were less busy. If they'd been using hard drives, restoration
would have been faster, but I can't imagine how big the
building for that would need to be (if all the hard drives
were actually on line at the same time).

Paul

On Mon, 11 Jun 2018 00:56:32 -0500, Char Jackson
wrote:

On Sun, 10 Jun 2018 16:53:09 -0000 (UTC), Peter Kozlov
wrote:

On 2018-06-10, Bennett wrote:
On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?

The bandwidth isn't there for something like that. I'm ingesting far too
much data for that to be an option at all. And the data isn't simple one
machine. It's non-machine specific.

There is a flow of large data-sets that arrive into machine A. And
machine A processes these into a share. From the share they are used by
a few different people. After a short while, the work is completed, but
the stays put for a while longer. This is happening multiple times per
day.

At the end of the day, everything new on A that doesn't exist on B is
pushed to B. As A becomes full, data-sets from A are deleted. They can
be deleted because there are backups for long term offline on B.

When B becomes full it is removed and stored for long-term. A new blank
B drive is attached and A is completely cleared. At this point, the
process starts all over again.

The daily amount of data can be is little as 9 GBs but that is rare.
It's typically between 150 GBs and 400 GBs. Sometimes it is much more. I
think I had to deal with 3 TBs one day. So the bandwidth I have to play
with is already stretched to the limites as it is. I can't afford to
allow something like Carbonite or any web-based solution because I need
all of my bandwidth. Much of what I am describing can come in over the
internet or on physical drives. As it is sometimes yesterday's work
bleeds into today, which is bad. That can start me off with new work
being delated while old work is completed. That drives me crazy.

What I need is a lot more bandwidth. And I need faster copies. This used
to be no more 50 GBs in a day and USB2 could be used for the backups
without too much trouble. It's not at a point where I am using SSDs on
much of this because that gives me about 329 MB/sec copy speed. Gigabit
Ethernet is quickly becoming a problem as well.

This isn't like backing a computer with small delta changes per day
that an ordinary user users. This is an ongoing archive or large
data-sets that flow through a media company. The footage is getting
larger and larger and resolution gets better and better. Image a 2 hour
movie in HD quality with something like ProRes. It's can easily be 220
GBs. And now 4k is a thing and 8k will follow. Imagine archiving show
after show after show in an endless number of on-going shows. Even on
very large arrays everything can't be kept on-line all the time. Things
have to be moved off at a pace to keep the flow flowing through. I can't
remove anything without having of a backup of it. And I have to give it
to other parties.

Your described use case is Machine A backs up to Machine B every day,
and when Machine B is full you swap it for a new (empty) Machine B,
right?

More specificaly, Drive A backs pushes all content not already on
Drive B on a constant basis. Drive B can be larger than Drive A. This
is because it is not a sync. Drive B simply gets anything it does not
have from Drive A. I can delete to save space on drive A and Drive B
is not affected.

When Drive B is replaced because it is full, Drive A must be erased
back to zero or close to zero otherwise, the next Drive B will get
everything Drive A has. Most of that will already be on the previous
Drive B that we just removed.

Drive A doesn't know what it has already copied to drive B or previous
Drive Bs.

Drive A is a live drive. It's actually a server share location.
Drive B is actually a physical USB 3 hard drive.

I think I would skip the second part of that and simply use Machine B as
a FIFO buffer, so rather than wait for B to be full and then swap it for
a new B, I would pull data off of B every evening, or as often as it
takes, so that as far as A is concerned, B is never full.

Regarding network connections, you have USB3, USB3.1, and of course
10Gig Ethernet if you're serious about speed. 10G NICs are fairly
affordable now, starting at about $30. But back to the paragraph above,
if B is a simple FIFO buffer, then you can constantly pull data off of B
while A is putting more data on. As far as A is concerned, B's capacity
is infinite.

Drive A (which is a server share) is the live data everyone is using
for a short period of time.

Drive B is a physcal USB driver.

One computer facilitates the copy process from A to B. Since A is a
network share and B is a physical drive, the network speed does play a
role in this. I have not yet upgraded to 10G. Absolutely need to do
that as soon as possible.

--
Peter Kozlov

On Mon, 11 Jun 2018 06:59:16 -0700, Peter Kozlov
wrote:

On Mon, 11 Jun 2018 00:56:32 -0500, Char Jackson
wrote:

On Sun, 10 Jun 2018 16:53:09 -0000 (UTC), Peter Kozlov
wrote:

On 2018-06-10, Bennett wrote:
On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.

What you are describing is an incremental backup. You can do this with
Macrium Reflect.

I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?

I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?

The bandwidth isn't there for something like that. I'm ingesting far too
much data for that to be an option at all. And the data isn't simple one
machine. It's non-machine specific.

There is a flow of large data-sets that arrive into machine A. And
machine A processes these into a share. From the share they are used by
a few different people. After a short while, the work is completed, but
the stays put for a while longer. This is happening multiple times per
day.

At the end of the day, everything new on A that doesn't exist on B is
pushed to B. As A becomes full, data-sets from A are deleted. They can
be deleted because there are backups for long term offline on B.

When B becomes full it is removed and stored for long-term. A new blank
B drive is attached and A is completely cleared. At this point, the
process starts all over again.

The daily amount of data can be is little as 9 GBs but that is rare.
It's typically between 150 GBs and 400 GBs. Sometimes it is much more. I
think I had to deal with 3 TBs one day. So the bandwidth I have to play
with is already stretched to the limites as it is. I can't afford to
allow something like Carbonite or any web-based solution because I need
all of my bandwidth. Much of what I am describing can come in over the
internet or on physical drives. As it is sometimes yesterday's work
bleeds into today, which is bad. That can start me off with new work
being delated while old work is completed. That drives me crazy.

What I need is a lot more bandwidth. And I need faster copies. This used
to be no more 50 GBs in a day and USB2 could be used for the backups
without too much trouble. It's not at a point where I am using SSDs on
much of this because that gives me about 329 MB/sec copy speed. Gigabit
Ethernet is quickly becoming a problem as well.

This isn't like backing a computer with small delta changes per day
that an ordinary user users. This is an ongoing archive or large
data-sets that flow through a media company. The footage is getting
larger and larger and resolution gets better and better. Image a 2 hour
movie in HD quality with something like ProRes. It's can easily be 220
GBs. And now 4k is a thing and 8k will follow. Imagine archiving show
after show after show in an endless number of on-going shows. Even on
very large arrays everything can't be kept on-line all the time. Things
have to be moved off at a pace to keep the flow flowing through. I can't
remove anything without having of a backup of it. And I have to give it
to other parties.

Your described use case is Machine A backs up to Machine B every day,
and when Machine B is full you swap it for a new (empty) Machine B,
right?

More specificaly, Drive A backs pushes all content not already on
Drive B on a constant basis. Drive B can be larger than Drive A. This
is because it is not a sync. Drive B simply gets anything it does not
have from Drive A. I can delete to save space on drive A and Drive B
is not affected.

When Drive B is replaced because it is full, Drive A must be erased
back to zero or close to zero otherwise, the next Drive B will get
everything Drive A has. Most of that will already be on the previous
Drive B that we just removed.

Drive A doesn't know what it has already copied to drive B or previous
Drive Bs.

Drive A is a live drive. It's actually a server share location.
Drive B is actually a physical USB 3 hard drive.

I think I would skip the second part of that and simply use Machine B as
a FIFO buffer, so rather than wait for B to be full and then swap it for
a new B, I would pull data off of B every evening, or as often as it
takes, so that as far as A is concerned, B is never full.

Regarding network connections, you have USB3, USB3.1, and of course
10Gig Ethernet if you're serious about speed. 10G NICs are fairly
affordable now, starting at about $30. But back to the paragraph above,
if B is a simple FIFO buffer, then you can constantly pull data off of B
while A is putting more data on. As far as A is concerned, B's capacity
is infinite.

Drive A (which is a server share) is the live data everyone is using
for a short period of time.

Drive B is a physcal USB driver.

One computer facilitates the copy process from A to B. Since A is a
network share and B is a physical drive, the network speed does play a
role in this. I have not yet upgraded to 10G. Absolutely need to do
that as soon as possible.

I ended up going with an app called, Visa Versa Pro. Seems to do what
I want without too much fuss. Problem is solved.

--
Peter Kozlov

Peter Kozlov wrote:
On Mon, 11 Jun 2018 00:56:32 -0500, Char Jackson
wrote:

On Sun, 10 Jun 2018 16:53:09 -0000 (UTC), Peter Kozlov
wrote:

On 2018-06-10, Bennett wrote:
On 6/9/2018 9:53 PM, Peter Kozlov wrote:
On 2018-06-10, Bennett wrote:
On 6/8/2018 7:57 PM, Peter Kozlov wrote:
On 2018-06-09, Stephen Chadfield wrote:
On Thu, 07 Jun 2018 13:54:00 -0700, Peter Kozlov
wrote:

Have any of you use Sync software?

Here's what I need to do.

I have data that grows daily on a server share. Not only does it grow,
but sometimes items are deleted. What I want is not truly a sync. What
I need is more like, compare A to B and if A has something B does not,
then copy A to B.

B is drive that grows and grows and when I fill it up, I replace B
with a new blank drive.

A is daiily work. At the end of the day, whatever is on A that isn't
on B needs to be copied over.

A is deleted whenever I need it cleared.

B is a running backup of all things that created daily on A.
What you are describing is an incremental backup. You can do this with
Macrium Reflect.
I just wanted this for files. Robocopy sounds better to me. I don't want
a back and restore method for this task. I just want software that looks
at A and checks to see if B has what A has and if not, it copies A to B.
If I delete from A I do not want this to delete from B. When B becomes
full, I just remove B and delete all that is on A and the process starts
all over again.

B is a record of everything that was ever added to A. If I deicde
something should not be on B, it will require my direct intervention to
remove it manually from B and A so that it doesn't get pushed from A to
B all over again.

B is just a simple removable hard drive. The contents of which are saved
to a text file. And if I need something, I will consult the text file to
see which drive has what I need. It's all pretty simple. No software
required to access to the data on B or A.

I'm curious as to what you do when B is full and the copy fails. It's
easy enough to deal with that manually but how do you automate it?
I deal with it manually. I've had to do it a few times, too.

What's the downside of using something like Carbonite or other real time
cloud based software?
The bandwidth isn't there for something like that. I'm ingesting far too
much data for that to be an option at all. And the data isn't simple one
machine. It's non-machine specific.

There is a flow of large data-sets that arrive into machine A. And
machine A processes these into a share. From the share they are used by
a few different people. After a short while, the work is completed, but
the stays put for a while longer. This is happening multiple times per
day.

At the end of the day, everything new on A that doesn't exist on B is
pushed to B. As A becomes full, data-sets from A are deleted. They can
be deleted because there are backups for long term offline on B.

When B becomes full it is removed and stored for long-term. A new blank
B drive is attached and A is completely cleared. At this point, the
process starts all over again.

The daily amount of data can be is little as 9 GBs but that is rare.
It's typically between 150 GBs and 400 GBs. Sometimes it is much more. I
think I had to deal with 3 TBs one day. So the bandwidth I have to play
with is already stretched to the limites as it is. I can't afford to
allow something like Carbonite or any web-based solution because I need
all of my bandwidth. Much of what I am describing can come in over the
internet or on physical drives. As it is sometimes yesterday's work
bleeds into today, which is bad. That can start me off with new work
being delated while old work is completed. That drives me crazy.

What I need is a lot more bandwidth. And I need faster copies. This used
to be no more 50 GBs in a day and USB2 could be used for the backups
without too much trouble. It's not at a point where I am using SSDs on
much of this because that gives me about 329 MB/sec copy speed. Gigabit
Ethernet is quickly becoming a problem as well.

This isn't like backing a computer with small delta changes per day
that an ordinary user users. This is an ongoing archive or large
data-sets that flow through a media company. The footage is getting
larger and larger and resolution gets better and better. Image a 2 hour
movie in HD quality with something like ProRes. It's can easily be 220
GBs. And now 4k is a thing and 8k will follow. Imagine archiving show
after show after show in an endless number of on-going shows. Even on
very large arrays everything can't be kept on-line all the time. Things
have to be moved off at a pace to keep the flow flowing through. I can't
remove anything without having of a backup of it. And I have to give it
to other parties.
Your described use case is Machine A backs up to Machine B every day,
and when Machine B is full you swap it for a new (empty) Machine B,
right?

More specificaly, Drive A backs pushes all content not already on
Drive B on a constant basis. Drive B can be larger than Drive A. This
is because it is not a sync. Drive B simply gets anything it does not
have from Drive A. I can delete to save space on drive A and Drive B
is not affected.

When Drive B is replaced because it is full, Drive A must be erased
back to zero or close to zero otherwise, the next Drive B will get
everything Drive A has. Most of that will already be on the previous
Drive B that we just removed.

Drive A doesn't know what it has already copied to drive B or previous
Drive Bs.

Drive A is a live drive. It's actually a server share location.
Drive B is actually a physical USB 3 hard drive.

I think I would skip the second part of that and simply use Machine B as
a FIFO buffer, so rather than wait for B to be full and then swap it for
a new B, I would pull data off of B every evening, or as often as it
takes, so that as far as A is concerned, B is never full.

Regarding network connections, you have USB3, USB3.1, and of course
10Gig Ethernet if you're serious about speed. 10G NICs are fairly
affordable now, starting at about $30. But back to the paragraph above,
if B is a simple FIFO buffer, then you can constantly pull data off of B
while A is putting more data on. As far as A is concerned, B's capacity
is infinite.

Drive A (which is a server share) is the live data everyone is using
for a short period of time.

Drive B is a physcal USB driver.

One computer facilitates the copy process from A to B. Since A is a
network share and B is a physical drive, the network speed does play a
role in this. I have not yet upgraded to 10G. Absolutely need to do
that as soon as possible.

There is an example of a card here. The x4 interface helps
get sufficient bandwidth to actually use it.

https://www.newegg.ca/Product/Produc...16833320272CVF

How it works, is hinted at here. Unlike USB3.1 Rev.2, it uses
a "fancy" scheme to transmit data on four pairs. It uses PAM16,
DSQ128, and a lot of what looks like DSP. Page 7 for example,
shows a representation of the cleanup of a received signal.
My guess is, the high power dissipation (5 to 8 watts)
for stuff like that, is because of the signal processing.
USB3.1 Rev.2 at 10Gbit/sec, is relatively naive by comparison
(and only supports short cabling), and as a result a USB3.1
device doesn't burn quite as much power.

http://www.ikn.no/download/Whitepape...rnet-10-08.pdf

The eye diagram for PAM-16 is here. You can see the eye is
wide open near the end of a symbol period. (By comparison,
regular GbE uses PAM-5 with the same sort of intentions.)

http://keydigital.be/images/wp_Choos...ooseCAT5_2.jpg

( http://keydigital.be/KnowledgeCenter...edPair_wp.html )

This is USB3.2 (two channels of 3.1 Rev2 bonded) on a scope
for comparison. No fancy eye diagram, just regular ones
and zeros in a plain format.

https://images.anandtech.com/doci/12...7996_575px.JPG

There's more than one I/O option for 10GbE, and lots of
opinions about what to use. (Fiber versus copper)

At one time, the cards were $500 a pop (and made by Intel),
but finally a worthy competitor has appeared. (You still
need to read the Newegg reviews though to see if
actual customers are happy with the product.)

https://en.wikipedia.org/wiki/Aquantia_Corporation

I still don't own any of those :-)

Using ICS and no 10GbE switch would reduce the install cost,
but wouldn't make using it as pleasant as I'd like.

https://en.wikipedia.org/wiki/Intern...ection_Sharing

Make sure to do thorough data integrity testing after
you get it set up. Checksum the file at the source end.
Transfer it to disk at the receiving end. Run a
checksum on the received file. Make sure they agree.

If you're using hard drives on Machine B, yes, you profit
by running faster than GbE to a ~200MB/sec single drive.
But if the 10GbE actually runs at the full rate, that's
1250MB/sec, and you could use a RAID array of hard drives
to keep up. As buying NVMe SSD drives just to keep up with
the I/O rate would be silly.

One thing SSD drives have going for them, is some
monstrously large ones have been produced. There
was a 40TB one in a 3.5" drive form factor, but it
costs as much as a new car, so doesn't make for
cost effective (or even reliable) long term storage.
The I/O rate on that one is so slow, you can write the
drive continuously at full (SATA) rate for five years,
without exceeding the P/E (program erase) cycle limit
on the flash.

As for hard drives, they come in helium filled and
regular (air-filled) drives. As I understand it,
the helium ones are only guaranteed to hold the
helium for five years. I don't know what happens
after that. You can get helium ones in sizes anywhere
from 6TB to 14TB or so. Helium allows more platters
per drive, which is one of the incentives. One thing
they're doing, is shaving down the thickness of the
platters too, to encourage really tight spacing
on platters. Yikes. The air filled ones run
warmer, if you stuff them like that. Regular
drives have a breather hole to equalize atmospheric
pressure (so the cover won't tin-can on you :-) ).

Paul