how to merge 2 directories keeping newer duplicates and all unique files

I want to merge two directories to create a new, third one.
Both source directories contain many files with distinct names,
but there are many duplicates. I want to combine the distinct
files and, of the dups, keep the newer one.

Can robocopy do this? I looked through the documentation and
it doesn't look like there can be more than one source and
one target directory.

On 7/2/20 6:49 PM, this is what Jason wrote:
I want to merge two directories to create a new, third one.
Both source directories contain many files with distinct names,
but there are many duplicates. I want to combine the distinct
files and, of the dups, keep the newer one.

Can robocopy do this? I looked through the documentation and
it doesn't look like there can be more than one source and
one target directory.

Well, copy all of directory one to a temp.
Now cut and paste the 2nd directory into temp. As windows finds the first dupe, you just say skip and remember for all.

Okay now you have all the unique files in temp, and dupes in dir #2.

Then I'd use some find dupe program to check those dupes in dir #2 with temp.

Meld? Finddupe? there are a ton of them. I've become quite fond of meld but it's great for text files, not so well with photos/images.

Al

On Thu, 2 Jul 2020 18:49:01 -0400, Jason
wrote:

I want to merge two directories to create a new, third one.
Both source directories contain many files with distinct names,
but there are many duplicates. I want to combine the distinct
files and, of the dups, keep the newer one.

Can robocopy do this? I looked through the documentation and
it doesn't look like there can be more than one source and
one target directory.

I had a similar problem a few months ago. I think you will find that
all you have to do is copy the contents into a new folder. If there
are duplicates Windows will give you several options of what to do.
From memory they include File Name + File Name (2), overwrite, do not
copy etc. It should be easy enough to set up a trial without losing
anything from the parent directories.
--

Regards,

Eric Stevens

Well, copy all of directory one to a temp.
Now cut and paste the 2nd directory into temp. As windows finds the first dupe, you just say skip and remember for all.

Okay now you have all the unique files in temp, and dupes in dir #2.

Then I'd use some find dupe program to check those dupes in dir #2 with temp.

Meld? Finddupe? there are a ton of them. I've become quite fond of meld but it's great for text files, not so well with photos/images.


I don't understand how that makes sure that the dupe with the
more current date gets selected.

On 7/2/2020 5:49 PM, Jason wrote:
I want to merge two directories to create a new, third one.
Both source directories contain many files with distinct names,
but there are many duplicates. I want to combine the distinct
files and, of the dups, keep the newer one.

Can robocopy do this? I looked through the documentation and
it doesn't look like there can be more than one source and
one target directory.

I don't know robocopy but the free version of SyncBackFree can do it.

I use it to keep a couple of directories syncronized between two PC's.
That way I can make updates on either machine and when the program runs
once a night, or however often I want to run it, files on both PC's are
compared and if both have the same name only the newest one is kept.

If a file is only on one PC but not the other it too will copy it where
needed. You could then delete the remaining duplicate folder if you wish.

On 7/2/20 10:52 PM, this is what Jason wrote:
Well, copy all of directory one to a temp.
Now cut and paste the 2nd directory into temp. As windows finds the first dupe, you just say skip and remember for all.

Okay now you have all the unique files in temp, and dupes in dir #2.

Then I'd use some find dupe program to check those dupes in dir #2 with temp.

Meld? Finddupe? there are a ton of them. I've become quite fond of meld but it's great for text files, not so well with photos/images.


I don't understand how that makes sure that the dupe with the
more current date gets selected.

The first copy will skip any dupes and just copy items that don't exist.
The 2nd part is inspecting what's left behind with your uniq items in temp dir and manually or automatically (depending on software) to find
which is newer.
It may even be part of a copy command with some switches to make it only copy newer items, I did not look at the options on copy. I run
Linux so it would be different anyway.

Al

Jason wrote:
I want to merge two directories to create a new, third one.
Both source directories contain many files with distinct names,
but there are many duplicates. I want to combine the distinct
files and, of the dups, keep the newer one.

Can robocopy do this? I looked through the documentation and
it doesn't look like there can be more than one source and
one target directory.


I had to let this one stew in the brain pan for a while. That's what
happens when the "major" arguments in robocopy, aren't described in
enough detail to stimulate the imagination.

robocopy a\ c\ /copy:datso /l # first, check what it would do
robocopy a\ c\ /copy:datso # Then run it for real

robocopy b\ c\ /copy:datso /l # Merge the b directory into c
robocopy b\ c\ /copy:datso # Newer files overwrite old, of the same name

a\
1.txt
2.txt
3.txt
4.txt
5.txt

b\
1.txt contents = "smokin" , save and exit makes it the newest file
2.txt contents = "fast"
6.txt
7.txt
8.txt

Results after the two (working) commands

c\
1.txt contents = "smokin"
2.txt contents = "fast"
3.txt
4.txt
5.txt
6.txt
7.txt
8.txt

HTH,
Paul

In article , lid says...

Jason wrote:
I want to merge two directories to create a new, third one.
Both source directories contain many files with distinct names,
but there are many duplicates. I want to combine the distinct
files and, of the dups, keep the newer one.

Can robocopy do this? I looked through the documentation and
it doesn't look like there can be more than one source and
one target directory.


I had to let this one stew in the brain pan for a while. That's what
happens when the "major" arguments in robocopy, aren't described in
enough detail to stimulate the imagination.

robocopy a\ c\ /copy:datso /l # first, check what it would do
robocopy a\ c\ /copy:datso # Then run it for real

robocopy b\ c\ /copy:datso /l # Merge the b directory into c
robocopy b\ c\ /copy:datso # Newer files overwrite old, of the same name

a\
1.txt
2.txt
3.txt
4.txt
5.txt

b\
1.txt contents = "smokin" , save and exit makes it the newest file
2.txt contents = "fast"
6.txt
7.txt
8.txt

Results after the two (working) commands

c\
1.txt contents = "smokin"
2.txt contents = "fast"
3.txt
4.txt
5.txt
6.txt
7.txt
8.txt

HTH,
Paul

There's not usually much you can add to Paul's posts - he's certainly someone I've learned to
recognise as one of the most valuable contributors here. But it's worth mentioning that
robocopy has an explicit option (/XO) which directs it not to copy files where the source is no
newer than the destination. I think that's the default behaviour anyway (?). Ah - it seems
not. I found this:

"If a file exists in both the source and destination locations, by default Robocopy copies the
file only if the two versions have different time stamps or different sizes. "

and..

"By default, Changed, Newer, and Older files are candidates for copying (subject to further
filtering, as described later). Same files are not copied."

So it's vital to use /XO - unless that default has changed in recent versions. Including /XO
will do no harm anyway.

These extracts are from the fearsomely detailed file "Robocopy.doc" which can be readily found
on the web. Note that it's a fairly old document, and there have been subtle changes to
robocopy in later versions of Windows (it's now built-in to W10). Robocopy is pretty unique in
my experience, and it can solve all sorts of complex file-grooming problems, so time spent
puzzling over the documentation is well-rewarded. Generally, I type Robocopy /? in a command
prompt, study the options, and look up any I'm unsure of in Robocopy.doc or other reference.

--

Phil, London

In article ,
says...


and..

"By default, Changed, Newer, and Older files are candidates for copying (subject to further
filtering, as described later). Same files are not copied."

So it's vital to use /XO - unless that default has changed in recent versions. Including /XO
will do no harm anyway.


Looks like that default is still the case. A relatively recent article:

https://adamtheautomator.com/robocop...-copying-files

"The most basic use of robocopy is using a source and destination directory with no options.
robocopy C:\src C:\dst
This option will copy all files (excluding subfolders) from C:\src to C:\dst."


--

Phil, London

Philip Herlihy wrote:
In article ,
says...

and..

"By default, Changed, Newer, and Older files are candidates for copying (subject to further
filtering, as described later). Same files are not copied."

So it's vital to use /XO - unless that default has changed in recent versions. Including /XO
will do no harm anyway.


Looks like that default is still the case. A relatively recent article:

https://adamtheautomator.com/robocop...-copying-files

"The most basic use of robocopy is using a source and destination directory with no options.
robocopy C:\src C:\dst
This option will copy all files (excluding subfolders) from C:\src to C:\dst."



I would have caught that, if I'd made a more comprehensive test case.

robocopy a\ c\ /copy:datso /xo /e /l # first, check what it would do
robocopy a\ c\ /copy:datso /xo /e # Then run it for real

robocopy b\ c\ /copy:datso /xo /e /l # Merge the b directory into c
robocopy b\ c\ /copy:datso /xo /e # Newer files overwrite old, of the same name
# xo = "exclude older" files as an overwrite
# e = recursive, include empty folders
a\
1.txt
2.txt
3.txt contents = "real" , save and exit makes it the newest file
4.txt contents = "soon" [for "old over new" test using b\]
5.txt
d\
9.txt

b\
1.txt contents = "smokin" , save and exit makes it the newest file
2.txt contents = "fast" [ for "new over old" test using b\ ]
3.txt
4.txt
6.txt
7.txt
8.txt
e\
10.txt

Results after the two (working) commands

c\
1.txt contents = "smokin"
2.txt contents = "fast"
3.txt contents = "real"
4.txt contents = "soon"
5.txt
6.txt
7.txt
8.txt
d\
9.txt
e\
10.txt

Paul

Jason,

Both source directories contain many files with distinct
names, but there are many duplicates. I want to combine
the distinct files and, of the dups, keep the newer one.

Does "xcopy" still exists on W10 ? If so :

/D copy only newer files

(use "xcopy /?" to get the whole list)

Regards,
Rudy Wieser

In article ,
says...
I don't know robocopy but the free version of SyncBackFree can do it.

Thanks!

In article ,
says...
These extracts are from the fearsomely detailed file "Robocopy.doc" which can be readily found
on the web.

Yep - I found that and it only amplified my confusion

On 7/3/2020 12:33 PM, R.Wieser wrote:
Jason,

Both source directories contain many files with distinct
names, but there are many duplicates. I want to combine
the distinct files and, of the dups, keep the newer one.

Does "xcopy" still exists on W10 ? If so :

/D copy only newer files

(use "xcopy /?" to get the whole list)

Regards,
Rudy Wieser


Nice one, I forgot about it, and it is still in Windows 10 too.

In article ,
says...

In article ,
says...
These extracts are from the fearsomely detailed file "Robocopy.doc" which can be readily found
on the web.

Yep - I found that and it only amplified my confusion

That document does separate the men from the boys! :-)

--

Phil, London