Robocopy & dates and such

Okay, for those new to the discussion:

I use Robocopy to sync files from the Harddrive (ntfs) to
thumbdrives (FAT32)
Saturday the 9th of March at 22:20 PST is started the sync and it
ended Sat Mar 09 22:21:48 2019, having copied all 13 files and all is
normal. Takes about one minutes.
Sunday evening, I run the same batchfile at 21:51 and it proceeds
to copy "everything"." From page 850 of the 2 logfile:
Start of transfers to Drive F was at 2151 "

Started loading Thumb at 2151
Ending Thumb Loading at (2237)
eot
This was, to say the least, very disconcerting.

The consensuses of the assemble minds here, is that it is a
feature of how the different file systems (NTFS and FAT32) handles the
minutia of time stamping files.

The suggestion I use the /TS switch is fine, but that merely logs
the time stamps of the files.
/CopyAT is default, it copys the Date Attributes and Time of the
source file.
adding the /TimFix switch, whatever it does, it is also copying
files as newer which had not been changed.

Ah-ha. It comes to mind "Is the Archive attribute bit set?" Yes,
it is! So use the /M switch to copy only those with the archive bit
set, and reset it. Testing it, that does seem to "fix" the issue of
copying the western world.

Why? I still do not know. This is one of those thing where the
layman's explanation is "Magic", and the Technical Explanation is
"pure freaking magic." B-)

--
pyotr filipivich
The question was asked: "Is Hindsight overrated?"
In retrospect, it appears to be.

On 3/12/19 8:51 PM, pyotr filipivich wrote:
Okay, for those new to the discussion:

I use Robocopy to sync files from the Harddrive (ntfs) to
thumbdrives (FAT32)
Saturday the 9th of March at 22:20 PST is started the sync and it
ended Sat Mar 09 22:21:48 2019, having copied all 13 files and all is
normal. Takes about one minutes.
Sunday evening, I run the same batchfile at 21:51 and it proceeds
to copy "everything"." From page 850 of the 2 logfile:
Start of transfers to Drive F was at 2151 "

Started loading Thumb at 2151
Ending Thumb Loading at (2237)
eot
This was, to say the least, very disconcerting.

The consensuses of the assemble minds here, is that it is a
feature of how the different file systems (NTFS and FAT32) handles the
minutia of time stamping files.

The suggestion I use the /TS switch is fine, but that merely logs
the time stamps of the files.
/CopyAT is default, it copys the Date Attributes and Time of the
source file.
adding the /TimFix switch, whatever it does, it is also copying
files as newer which had not been changed.

Ah-ha. It comes to mind "Is the Archive attribute bit set?" Yes,
it is! So use the /M switch to copy only those with the archive bit
set, and reset it. Testing it, that does seem to "fix" the issue of
copying the western world.

Why? I still do not know. This is one of those thing where the
layman's explanation is "Magic", and the Technical Explanation is
"pure freaking magic." B-)

Did you try the /DT switch?

On 3/12/19 11:09 PM, Big Al wrote:
On 3/12/19 8:51 PM, pyotr filipivich wrote:
Okay, for those new to the discussion:

Â*Â*Â*Â*I use Robocopy to sync files from the Harddrive (ntfs) to
thumbdrives (FAT32)
Â*Â*Â*Â*Saturday the 9th of March at 22:20 PST is started the sync and it
ended Sat Mar 09 22:21:48 2019, having copied all 13 files and all is
normal. Takes about one minutes.
Â*Â*Â*Â*Sunday evening, I run the same batchfile at 21:51 and it proceeds
to copy "everything"."Â* From page 850 of the 2 logfile:
Start of transfers to Drive F was at 2151 "

Â* Started loading Thumb at 2151
Â*Â*Â* Ending Thumb Loading at (2237)
eot
Â*Â*Â*Â*This was, to say the least, very disconcerting.

Â*Â*Â*Â*The consensuses of the assemble minds here, is that it is a
feature of how the different file systems (NTFS and FAT32) handles the
minutia of time stamping files.

Â*Â*Â*Â*The suggestion I use the /TS switch is fine, but that merely logs
the time stamps of the files.
Â*Â*Â*Â*/CopyAT is default, it copys the Date Attributes and Time of the
source file.
Â*Â*Â*Â*adding the /TimFix switch, whatever it does, it is also copying
files as newer which had not been changed.

Â*Â*Â*Â*Ah-ha.Â* It comes to mind "Is the Archive attribute bit set?" Yes,
it is!Â* So use the /M switch to copy only those with the archive bit
set, and reset it.Â* Testing it, that does seem to "fix" the issue of
copying the western world.

Â*Â*Â*Â*Why?Â* I still do not know.Â* This is one of those thing where the
layman's explanation is "Magic", and the Technical Explanation is
"pure freaking magic."Â* B-)

Did you try the /DT switch?

Sorry that's /DST?

Big Al wrote:

On 3/12/19 11:09 PM, Big Al wrote:
On 3/12/19 8:51 PM, pyotr filipivich wrote:
Okay, for those new to the discussion:

****I use Robocopy to sync files from the Harddrive (ntfs) to
thumbdrives (FAT32)
****Saturday the 9th of March at 22:20 PST is started the sync and it
ended Sat Mar 09 22:21:48 2019, having copied all 13 files and all is
normal. Takes about one minutes.
****Sunday evening, I run the same batchfile at 21:51 and it proceeds
to copy "everything"."* From page 850 of the 2 logfile:
Start of transfers to Drive F was at 2151 "

* Started loading Thumb at 2151
*** Ending Thumb Loading at (2237)
eot
****This was, to say the least, very disconcerting.

****The consensuses of the assemble minds here, is that it is a
feature of how the different file systems (NTFS and FAT32) handles the
minutia of time stamping files.

****The suggestion I use the /TS switch is fine, but that merely logs
the time stamps of the files.
****/CopyAT is default, it copys the Date Attributes and Time of the
source file.
****adding the /TimFix switch, whatever it does, it is also copying
files as newer which had not been changed.

****Ah-ha.* It comes to mind "Is the Archive attribute bit set?" Yes,
it is!* So use the /M switch to copy only those with the archive bit
set, and reset it.* Testing it, that does seem to "fix" the issue of
copying the western world.

****Why?* I still do not know.* This is one of those thing where the
layman's explanation is "Magic", and the Technical Explanation is
"pure freaking magic."* B-)

Did you try the /DT switch?

Sorry that's /DST?

And until you get the options correct (assuming the datestamp, a long
interger versus just hours & minutes), the OP might want to use the /L
command-line switch to see just what robocopy intends to do.

If the OP is using /DCOPY (mistaking it for /COPY), the defaults for
/DCOPY are D (Data) and A (Archive) for what to copy from source to
destination. The source timestamps are not copied to the destination
files for /DCOPY, by default, so the destination files should get a new
date for the newly created files. As I recall, /DCOPY will retain the
timestamps for the folders wherease /COPY might not (it only preserves
the file timestamps) -- this is from memory and what I read, so it would
needs to get tested. If the OP was specifying a directory as the source
or destination arguments when using /COPY, I suppose the directory
timestamps could be different while the file timestamps were the same;
that is, with directories specified instead of filespecs, the filenames
might have the same timestamps but not the directories.

I think Paul might been onto something with the differences in
timestamping granularity in file systems between the source and
destination. NTFS retains timestamps down to a resolution of 100
nanoseconds (0.0000001 s). FAT32 stores only at a resolution of 2
seconds. That is a 2 billion difference in granularity. You can use
wmic to retrieve a file's timestamps down to the nearest microsecond.

wmic datafile where name='c:\\temp\\testme.txt' get CreationDate

This shows the Creation Date as an decimal value in the format
yyyymmddHHMMSS.xxxxxxsUUU, whe

yyyy = 4-digit year (0000 - 9999)
mm = 2-digit month (01 - 12)
dd = 2-digit day of month (01 - 31)
HH = 2-digit hour (00 - 23)
MM = 2-digit minutes (00 - 59)
SS = 2-digit seconds (00 - 59)
.xxxxxx = decimal seconds down to microsecond (000000 - 999999)
s = Negative or positive (- or +) offset from Coordinated
Universal Time.
UUU = 3-digit offset in minutes originating timezone deviates from
Coordinated Universal Time.

You need to remove the space character in the property names for the
'get' command. Use CreationDate, not Creation Date. Double backslashes
are need in the name field: the first backslash escapes the next
backslash which is otherwise a special character. In my example, I'm
looking at the Creation Date value for a file in c:\temp\testme.txt.

Since robocopy is comparing last-modified timestamps, you would use:

wmic datafile where name='path\\filename' get LastModified

I ran these commands under Windows 7 for a file stored in an NTFS
partition. I didn't have a FAT32 partition to test what value gets
reported by wmic for the Create or Last Modified datestamps.

I'm wondering with the change for daylight savings if the UUU value
might've changed making the source files look an hour different than
existing destination files by the same name. The destination files were
created earlier, the daylight change happened, and now the source files
look an hour off from the offset for the prior destination files. From
what I've read, this problem only occurs for FAT32 where Windows does
not automatically update the timestamps for DST changes. I doubt
Windows rewrites all files to affect a 60-minute change to the UUU value
for every file but instead just computes the offset for the system clock
to bias the UUU value.

https://freefilesync.org/manual.php?...ht-saving-time
"NTFS stores time in UTC format, while FAT uses local time."

To eliminate the extraneous all-file copy by robocopy twice per year for
each DST change, the OP would need to convert his USB drive from FAT32
to NTFS. Then the files on his USB drive would be compared on their UTC
time rather than a local time that changes twice per yead due to DST.

Big Al on Tue, 12 Mar 2019 23:09:43 -0400 typed
in alt.windows7.general the following:
On 3/12/19 8:51 PM, pyotr filipivich wrote:
Okay, for those new to the discussion:

I use Robocopy to sync files from the Harddrive (ntfs) to
thumbdrives (FAT32)
Saturday the 9th of March at 22:20 PST is started the sync and it
ended Sat Mar 09 22:21:48 2019, having copied all 13 files and all is
normal. Takes about one minutes.
Sunday evening, I run the same batchfile at 21:51 and it proceeds
to copy "everything"." From page 850 of the 2 logfile:
Start of transfers to Drive F was at 2151 "

Started loading Thumb at 2151
Ending Thumb Loading at (2237)
eot
This was, to say the least, very disconcerting.

The consensuses of the assemble minds here, is that it is a
feature of how the different file systems (NTFS and FAT32) handles the
minutia of time stamping files.

The suggestion I use the /TS switch is fine, but that merely logs
the time stamps of the files.
/CopyAT is default, it copys the Date Attributes and Time of the
source file.
adding the /TimFix switch, whatever it does, it is also copying
files as newer which had not been changed.

Ah-ha. It comes to mind "Is the Archive attribute bit set?" Yes,
it is! So use the /M switch to copy only those with the archive bit
set, and reset it. Testing it, that does seem to "fix" the issue of
copying the western world.

Why? I still do not know. This is one of those thing where the
layman's explanation is "Magic", and the Technical Explanation is
"pure freaking magic." B-)

Did you try the /DT switch?

Yep.
--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

VanguardLH on Wed, 13 Mar 2019 02:02:48 -0500 typed in
alt.windows7.general the following:

I think Paul might been onto something with the differences in
timestamping granularity in file systems between the source and
destination. NTFS retains timestamps down to a resolution of 100
nanoseconds (0.0000001 s). FAT32 stores only at a resolution of 2
seconds. That is a 2 billion difference in granularity. You can use
wmic to retrieve a file's timestamps down to the nearest microsecond.

Now my question is: how does that impact where a file hasn't been
accessed in (2019-1996=.... carry the one,) over 23 years?
Is it a matter that subtracting the target date from the source
date, the source date is "effectively" a fraction of a second less
than the target date and is considered "newer" by at least 0.0000001
second? As I've said before "Makes sense. Not to me, but I'm sure
it does to someone."
--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

Big Al on Tue, 12 Mar 2019 23:09:43 -0400 typed
in alt.windows7.general the following:
On 3/12/19 8:51 PM, pyotr filipivich wrote:
Okay, for those new to the discussion:

I use Robocopy to sync files from the Harddrive (ntfs) to
thumbdrives (FAT32)
Saturday the 9th of March at 22:20 PST is started the sync and it
ended Sat Mar 09 22:21:48 2019, having copied all 13 files and all is
normal. Takes about one minutes.
Sunday evening, I run the same batchfile at 21:51 and it proceeds
to copy "everything"." From page 850 of the 2 logfile:
Start of transfers to Drive F was at 2151 "

Started loading Thumb at 2151
Ending Thumb Loading at (2237)
eot
This was, to say the least, very disconcerting.

The consensuses of the assemble minds here, is that it is a
feature of how the different file systems (NTFS and FAT32) handles the
minutia of time stamping files.

The suggestion I use the /TS switch is fine, but that merely logs
the time stamps of the files.
/CopyAT is default, it copys the Date Attributes and Time of the
source file.
adding the /TimFix switch, whatever it does, it is also copying
files as newer which had not been changed.

Ah-ha. It comes to mind "Is the Archive attribute bit set?" Yes,
it is! So use the /M switch to copy only those with the archive bit
set, and reset it. Testing it, that does seem to "fix" the issue of
copying the western world.

Why? I still do not know. This is one of those thing where the
layman's explanation is "Magic", and the Technical Explanation is
"pure freaking magic." B-)

Did you try the /DT switch?

/DST for daylight savings time? I missed that the first go round.
"fortunately", I still have "spare" sync files to test with.
--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

VanguardLH on Wed, 13 Mar 2019 02:02:48 -0500 typed in
alt.windows7.general the following:

wmic datafile where name='c:\\temp\\testme.txt' get CreationDate

Been using that to get current time. Looks like I will need to
spend some time poking at it to see what it does. Whoo-hoo. (now,
let see what I can give up for Lent...)

pyotr
--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

pyotr filipivich wrote:

Now my question is: how does that impact where a file hasn't been
accessed in (2019-1996=.... carry the one,) over 23 years?

NTFS uses UTC, so a change for DST doesn't affect the recorded
timestamp. FAT32 uses the local time, and you change that for DST. You
are using the datestamp for the source file on NTFS against the
datestamp for the destination file on FAT32, and Windows is using the
local time for the FAT32 partition's files.

The article at the end of my length reply describes why FAT32 sucks for
file comparison on datestamp under Windows due to the change for DST.

VanguardLH on Wed, 13 Mar 2019 20:03:56 -0500 typed in
alt.windows7.general the following:
pyotr filipivich wrote:

Now my question is: how does that impact where a file hasn't been
accessed in (2019-1996=.... carry the one,) over 23 years?

NTFS uses UTC, so a change for DST doesn't affect the recorded
timestamp. FAT32 uses the local time, and you change that for DST. You
are using the datestamp for the source file on NTFS against the
datestamp for the destination file on FAT32, and Windows is using the
local time for the FAT32 partition's files.

The article at the end of my length reply describes why FAT32 sucks for
file comparison on datestamp under Windows due to the change for DST.

Thanks.
--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

On Wed, 13 Mar 2019 02:02:48 -0500, VanguardLH wrote:
https://freefilesync.org/manual.php?...ht-saving-time
"NTFS stores time in UTC format, while FAT uses local time."

They have the general idea right, but their "solution" of formatting
the USB stick as NTFS is ridiculous, and can reduce its life. Here's
one reference:

https://www.howtogeek.com/177529/htg...are-removable-
drives-still-using-fat32-instead-of-ntfs/

To eliminate the extraneous all-file copy by robocopy twice per year for
each DST change, the OP would need to convert his USB drive from FAT32
to NTFS.

Or just use the /DST option in Robocopy. As I posted separately, that
worked for me in general, though there were four files, each several
years old, that it wanted to copy after the DST change, though it had
not wanted to copy them before the change.

As someone else posted, always run Robocopy with /L first, if you're
not sure what your chosen options will do.


--
Stan Brown, Oak Road Systems, Tompkins County, New York, USA
http://BrownMath.com/
http://OakRoadSystems.com/
Shikata ga nai...

Stan Brown on Thu, 14 Mar 2019 16:17:04
-0400 typed in alt.windows7.general the following:
On Wed, 13 Mar 2019 02:02:48 -0500, VanguardLH wrote:
https://freefilesync.org/manual.php?...ht-saving-time
"NTFS stores time in UTC format, while FAT uses local time."

They have the general idea right, but their "solution" of formatting
the USB stick as NTFS is ridiculous, and can reduce its life. Here's
one reference:

https://www.howtogeek.com/177529/htg...are-removable-
drives-still-using-fat32-instead-of-ntfs/

To eliminate the extraneous all-file copy by robocopy twice per year for
each DST change, the OP would need to convert his USB drive from FAT32
to NTFS.

Or just use the /DST option in Robocopy. As I posted separately, that
worked for me in general, though there were four files, each several
years old, that it wanted to copy after the DST change, though it had
not wanted to copy them before the change.

As someone else posted, always run Robocopy with /L first, if you're
not sure what your chosen options will do.

I knew what it would do, ore was suppose to do. I was just vastly
surprised at what it did _this_ time.

But I shall play with it "anon".
--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?