Macrium scheduling

Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful advice here. After
a few months of occasional checks I left it to do its stuff and essentially forgot
about it. It was scheduled very early every Saturday morning so I didn't often get
to see it start anyway. But yesterday I discovered that every weekly run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my destination
external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice of various
scheduling templates, I think my previous was 'Incremental Backup Set', and that
now seems perhaps a bit over complicated for a solo end user with no network stuff?
I'd appreciate others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly Full
("Grandfather") backups.

#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential backups.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental backups.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever creating a single
Full backup.

After this Incremental backups are created ad infinitum. The Full backup is
consolidated with subsequent Incremental backups once the specified number of
Incremental backups is reached.

This is also known as a Synthetic Full backup.

--------------------

As well as the 'method', any tips on optimal sequence of the full and incremental
runs?

Oh, and how often do others make a new recovery USB?


Terry, East Grinstead, UK

On 11/19/19 6:27 AM, Terry Pinnell wrote:
Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful advice here. After
a few months of occasional checks I left it to do its stuff and essentially forgot
about it. It was scheduled very early every Saturday morning so I didn't often get
to see it start anyway. But yesterday I discovered that every weekly run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my destination
external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice of various
scheduling templates, I think my previous was 'Incremental Backup Set', and that
now seems perhaps a bit over complicated for a solo end user with no network stuff?
I'd appreciate others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly Full
("Grandfather") backups.

#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential backups.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental backups.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever creating a single
Full backup.

After this Incremental backups are created ad infinitum. The Full backup is
consolidated with subsequent Incremental backups once the specified number of
Incremental backups is reached.

This is also known as a Synthetic Full backup.

--------------------

As well as the 'method', any tips on optimal sequence of the full and incremental
runs?

Oh, and how often do others make a new recovery USB?


Terry, East Grinstead, UK

I do full backups. HD space is cheap and I have plenty. My backup is
40GB and I can get 5 easily on my 1TB drive. I find it easier if I
have to reload to know that I don't have to jump through hoops to get a
proper load.

Truthfully I'm not really sure how incremental restores work. Do you
have to restore A then B the C etc or does the software do it
automatically? For this reason I just opted for full every time. I
guess I opted for the lazy way out and not do research.

I use Acronis so the restore of incremental may be different than
Macrium. With Acronis I can restore a single file or single partition
even though its a full backup. So I have the flexibility I need.

Al

Terry Pinnell wrote:
Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful advice here. After
a few months of occasional checks I left it to do its stuff and essentially forgot
about it. It was scheduled very early every Saturday morning so I didn't often get
to see it start anyway. But yesterday I discovered that every weekly run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my destination
external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice of various
scheduling templates, I think my previous was 'Incremental Backup Set', and that
now seems perhaps a bit over complicated for a solo end user with no network stuff?
I'd appreciate others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly Full
("Grandfather") backups.

#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential backups.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental backups.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever creating a single
Full backup.

After this Incremental backups are created ad infinitum. The Full backup is
consolidated with subsequent Incremental backups once the specified number of
Incremental backups is reached.

This is also known as a Synthetic Full backup.

--------------------

As well as the 'method', any tips on optimal sequence of the full and incremental
runs?

Oh, and how often do others make a new recovery USB?


Terry, East Grinstead, UK

Since Uwe was along today, in his honor we can promote this program.
It's a Drive Letter Manager to help with USB devices.

https://www.uwe-sieber.de/usbdlm_e.html

*******

#2 and #3 are part of implementing #1.

#4 is a fancy version of #3.

So that leaves a discussion of #1.

#1 starts with a Full, followed by incrementals.
You can have as many incrementals as you want, but
they have to remain readable. If you have 200 of them,
and the third one croaks, you have a large pile of unusable
incrementals and a very poor outcome. The incremental
chain must be truncated, by doing a Full every once in
a while. Then the incrementals start again, being applied
to that Full. This is a reliability issue. If you keep
200 Incrementals with respect to a Full, all 200 have to be
readable when you "restore to the 200th one".

Macrium (as it should), treats a Full and a set of Incrementals
as a "complete set". If you need to free up space, it generally
deletes a Full plus the associated Incrementals, as a set.

So what does the "Father" do ? since it's a differential,
it works out the differences since the original Full. It
allows the incrementals to be deleted, and more incrementals
to be run. At the end of the week, another Differential
is created, and the incrementals erased (since the Differential
captured everything they did, and now the Incrementals
are redundant).

The Differentials are points in time, so if we need to
go back a week, we can. The incrementals give us one day
resolution (but only for the last week). While the
differentials allow us to go back a week at a time only.

When the end of the month arrives, a Full is made. This
allows the previous Full and the Differentials to be erased
if space is needed. The Fulls collected each month, allow
us to go back a month for each one. But they're likely to be
large files, so keeping 12 of those would be a handful. At
some point, the Capacity Manager thing is going to start
throwing month fulls, overboard. If I did that sequence
on this machine, my backup drive would be exhausted by
January with that pattern. The incrementals and weekly
differentials wouldn't be that big of a burden, but collecting
Fulls once a month will add up. (Twelve times 2TB would be
24TB per year.)

The Incrementals Forever idea, is to have the efficiency of
Incrementals (really short backup interval), with the capping off
using a synthetic full, for some period of time. But by itself,
#4 isn't a complete strategy, whereas #1 is closer to a complete
strategy (a strategy the IT department might use, with some
adjustments).

Once the monthly Fulls are made by #1, you have the option
of moving them off the drive to make room for the pattern.
For example, since my backup drive could not handle too many
Fulls collected once a month, when I hit January, I might move
November to another drive. Leaving December and January on
there. If I were to delete the current Full, then all the
associated Incrementals and Differentials should disappear at
the same instant. Whereas if I throw away the July Full,
it is independent of what is happening in November. If you're
going to adopt #1, I'd probably want at least room for two
Fulls, and then move the excess (old) Fulls somewhere else.

Once you have 24 Fulls in hand, you could keep the oldest one
and the twelth one, and those are your yearly backups. Then
keep monthly ones that the sequence is generating. For example,
I once had to go back two years, to discover when my drive got
a certain "unremovable driver" added to it. So old backups
come in handy, "once in a lifetime". If I only kept those with
a temporal resolution of "once a year", I probably wouldn't be
that upset.

So while #1 is the correct sequence, it hasn't made allowance
for a yearly plan as such. You can do that manually of course.
You can add more "tiers" to the plan, and make it a bit more
sensible with respect to the size of the backup drive. For
example, I don't really need monthly ones forever, and
after three months, I could keep ones from every quarter
and from every year after that. I would need some kind of
RAID array, to make a pool big enough for #1 without modification :-)
If you had a dinky C: and a huge external, then perhaps #1
could run for quite a while without intervention.

Paul

Big Al wrote:
On 11/19/19 6:27 AM, Terry Pinnell wrote:
Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful
advice here. After
a few months of occasional checks I left it to do its stuff and
essentially forgot
about it. It was scheduled very early every Saturday morning so I
didn't often get
to see it start anyway. But yesterday I discovered that every weekly
run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my
destination
external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice
of various
scheduling templates, I think my previous was 'Incremental Backup
Set', and that
now seems perhaps a bit over complicated for a solo end user with no
network stuff?
I'd appreciate others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly
Full
("Grandfather") backups.

#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential
backups.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental
backups.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever
creating a single
Full backup.

After this Incremental backups are created ad infinitum. The Full
backup is
consolidated with subsequent Incremental backups once the specified
number of
Incremental backups is reached.

This is also known as a Synthetic Full backup.

--------------------

As well as the 'method', any tips on optimal sequence of the full and
incremental
runs?

Oh, and how often do others make a new recovery USB?


Terry, East Grinstead, UK

I do full backups. HD space is cheap and I have plenty. My backup is
40GB and I can get 5 easily on my 1TB drive. I find it easier if I
have to reload to know that I don't have to jump through hoops to get a
proper load.

Truthfully I'm not really sure how incremental restores work. Do you
have to restore A then B the C etc or does the software do it
automatically? For this reason I just opted for full every time. I
guess I opted for the lazy way out and not do research.

I use Acronis so the restore of incremental may be different than
Macrium. With Acronis I can restore a single file or single partition
even though its a full backup. So I have the flexibility I need.

Al

One full plus one differential, restores a differential to the desired date.
At any time, all we need is the two files, to have integrity. But, the
differential scheme wastes space (compared to incremental).

One full plus *every* incremental to the desired date, gets you back
to the desired date. If you chain too many incrementals together,
the conditional probability of a "problem" with just one of them,
becomes too great for a mathematical notion of reliability. If you
chain 200 of them together, and "try to restore to today", one of
those is bound to have a "read error" and then you're cooked with
respect to restoring to today. You shouldn't set the incremental
pool size too large, as a result.

One could argue you're in just as much danger, by using
one and only one backup drive connected externally to the
machine, since that has everything on it, and is a single
point of failure. Moving files off it, or duplicating it,
might make a purist feel better. Duplicating the backup drive,
has as much to do with "protecting you from your own stupidity",
as anything related to hardware. Clicking the wrong button in
the backup software, might knock out a set of backups. Or the
Capacity Manager might erase things that you decide later, you
would rather have kept. You can either move the stuff off, that
is safe to move (the bits that won't affect the current
state machine), or you can duplicate the whole thing once
in a while. And leave the second device disconnected, and sitting
in the corner for emergencies.

The possibilities are endless... and expensive.

Paul

On 11/19/19 8:50 AM, Paul wrote:
YouÂ*canÂ*eitherÂ*moveÂ*theÂ*stuffÂ*off,Â*that
isÂ*safeÂ*toÂ*moveÂ*(theÂ*bitsÂ*thatÂ*won'tÂ*affec tÂ*theÂ*current
stateÂ*machine),Â*orÂ*youÂ*canÂ*duplicateÂ*theÂ*wh oleÂ*thingÂ*once
inÂ*aÂ*while.Â*AndÂ*leaveÂ*theÂ*secondÂ*deviceÂ*di sconnected,Â*andÂ*sitting
inÂ*theÂ*cornerÂ*forÂ*emergencies.
I keep a 2nd copy on a 2nd drive of all my backups.
I used to work in a work environment where we pulled the tape (yes tape
drive, that's how long ago it was) out of the machine in the morning and
placed it in one of the secretaries purses to take home. Always one
off site.

Al

Big Al wrote:

On 11/19/19 8:50 AM, Paul wrote:
You*can*either*move*the*stuff*off,*that
is*safe*to*move*(the*bits*that*won't*affect*the*cu rrent
state*machine),*or*you*can*duplicate*the*whole*thi ng*once
in*a*while.*And*leave*the*second*device*disconnect ed,*and*sitting
in*the*corner*for*emergencies.
I keep a 2nd copy on a 2nd drive of all my backups.
I used to work in a work environment where we pulled the tape (yes tape
drive, that's how long ago it was) out of the machine in the morning and
placed it in one of the secretaries purses to take home. Always one
off site.

Al

Thanks Al. I'm tempted to do the same, largely because I can understand what's going
on more easily! But I'm going to have another coffee and read Paul's comprehensive
response at least once more before I decide.

P.S. The tapes I can remember from my training days with IBM would need *very* large
purses ;-)

Terry, East Grinstead, UK

Paul wrote:

Terry Pinnell wrote:
Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful advice here. After
a few months of occasional checks I left it to do its stuff and essentially forgot
about it. It was scheduled very early every Saturday morning so I didn't often get
to see it start anyway. But yesterday I discovered that every weekly run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my destination
external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice of various
scheduling templates, I think my previous was 'Incremental Backup Set', and that
now seems perhaps a bit over complicated for a solo end user with no network stuff?
I'd appreciate others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly Full
("Grandfather") backups.

#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential backups.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental backups.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever creating a single
Full backup.

After this Incremental backups are created ad infinitum. The Full backup is
consolidated with subsequent Incremental backups once the specified number of
Incremental backups is reached.

This is also known as a Synthetic Full backup.

--------------------

As well as the 'method', any tips on optimal sequence of the full and incremental
runs?

Oh, and how often do others make a new recovery USB?


Terry, East Grinstead, UK

Since Uwe was along today, in his honor we can promote this program.
It's a Drive Letter Manager to help with USB devices.

https://www.uwe-sieber.de/usbdlm_e.html

*******

#2 and #3 are part of implementing #1.

#4 is a fancy version of #3.

So that leaves a discussion of #1.

#1 starts with a Full, followed by incrementals.
You can have as many incrementals as you want, but
they have to remain readable. If you have 200 of them,
and the third one croaks, you have a large pile of unusable
incrementals and a very poor outcome. The incremental
chain must be truncated, by doing a Full every once in
a while. Then the incrementals start again, being applied
to that Full. This is a reliability issue. If you keep
200 Incrementals with respect to a Full, all 200 have to be
readable when you "restore to the 200th one".

Macrium (as it should), treats a Full and a set of Incrementals
as a "complete set". If you need to free up space, it generally
deletes a Full plus the associated Incrementals, as a set.

So what does the "Father" do ? since it's a differential,
it works out the differences since the original Full. It
allows the incrementals to be deleted, and more incrementals
to be run. At the end of the week, another Differential
is created, and the incrementals erased (since the Differential
captured everything they did, and now the Incrementals
are redundant).

The Differentials are points in time, so if we need to
go back a week, we can. The incrementals give us one day
resolution (but only for the last week). While the
differentials allow us to go back a week at a time only.

When the end of the month arrives, a Full is made. This
allows the previous Full and the Differentials to be erased
if space is needed. The Fulls collected each month, allow
us to go back a month for each one. But they're likely to be
large files, so keeping 12 of those would be a handful. At
some point, the Capacity Manager thing is going to start
throwing month fulls, overboard. If I did that sequence
on this machine, my backup drive would be exhausted by
January with that pattern. The incrementals and weekly
differentials wouldn't be that big of a burden, but collecting
Fulls once a month will add up. (Twelve times 2TB would be
24TB per year.)

The Incrementals Forever idea, is to have the efficiency of
Incrementals (really short backup interval), with the capping off
using a synthetic full, for some period of time. But by itself,
#4 isn't a complete strategy, whereas #1 is closer to a complete
strategy (a strategy the IT department might use, with some
adjustments).

Once the monthly Fulls are made by #1, you have the option
of moving them off the drive to make room for the pattern.
For example, since my backup drive could not handle too many
Fulls collected once a month, when I hit January, I might move
November to another drive. Leaving December and January on
there. If I were to delete the current Full, then all the
associated Incrementals and Differentials should disappear at
the same instant. Whereas if I throw away the July Full,
it is independent of what is happening in November. If you're
going to adopt #1, I'd probably want at least room for two
Fulls, and then move the excess (old) Fulls somewhere else.

Once you have 24 Fulls in hand, you could keep the oldest one
and the twelth one, and those are your yearly backups. Then
keep monthly ones that the sequence is generating. For example,
I once had to go back two years, to discover when my drive got
a certain "unremovable driver" added to it. So old backups
come in handy, "once in a lifetime". If I only kept those with
a temporal resolution of "once a year", I probably wouldn't be
that upset.

So while #1 is the correct sequence, it hasn't made allowance
for a yearly plan as such. You can do that manually of course.
You can add more "tiers" to the plan, and make it a bit more
sensible with respect to the size of the backup drive. For
example, I don't really need monthly ones forever, and
after three months, I could keep ones from every quarter
and from every year after that. I would need some kind of
RAID array, to make a pool big enough for #1 without modification :-)
If you had a dinky C: and a huge external, then perhaps #1
could run for quite a while without intervention.

Paul


Great post, thanks Paul. It should get a sticky in the Macrium forum if it doesn't
have one already.

As per my reply to Al, simplicity weighs heavily in my thinking. Particularly if
that unsought day arises and I have to restore an image. Can I really be confident
of navigating safely to what seems a rather complex 'set' of G-F-S files and
faithfully restoring them? If I'd already done it once then probably Yes, but...

And I also seriously doubt I'd ever want to restore a state more than a month or
three ago, given the frequent changes I make, plus WU's. While not in your terabyte
league, capacity isn't a trivial factor here; last image was over 150 GB.

I'll sleep on it for a few nights.

--------------------

I was about to implement the automatic failure email feature, but see it needs an
upgrade. I'll probably do that, but meanwhile maybe write a simple macro. It would
just open Macrium soon after each run, click the Log tab, expand the top entry,
check for a red cross, and display a prominent message.

--------------------

Will also look at that handy drive lettering tool you suggested.


Terry, East Grinstead, UK

Terry Pinnell wrote:

Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful
advice here. After a few months of occasional checks I left it to do
its stuff and essentially forgot about it. It was scheduled very
early every Saturday morning so I didn't often get to see it start
anyway. But yesterday I discovered that every weekly run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my
destination external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice
of various scheduling templates, I think my previous was
'Incremental Backup Set', and that now seems perhaps a bit over
complicated for a solo end user with no network stuff? I'd appreciate
others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly
Full ("Grandfather") backups.

Reflect doesn't do true GFS backups where, for example, a grandfather
father and son scheduled at the same time would only have the
grandfather backup run. I haven't found a consumer-grade backup program
that does a true GFS. Instead they just use the GFS concept for
scheduling. If you schedule them at the same time, you don't which will
run. If you schedule the grandfather with the father starting 5 minutes
later and the son starting 10 minutes later, you know the order they
will run: grandfather, father, and then son. Reflect will only run one
backup at a time, so the father pends until the grandfather finishes,
and the son pends until the father finishes.

That I did was to schedule the grandfather backup (full) to run on the
first Saturday of the month, the father backup (differential) on Sunday
(at the same time but different day as the grandfather), and the son
backup (incremental) to start 5 minutes later on every day. I could
have grandfather run on the first Saturday, father on the first Saturday
but 5 minutes later, and the son on every day 10 minutes later, too.
Since the grandfather just ran, the father would be very small. Not
much changed between when the grandfather and father backups ran, and
the father backup is mostly the size of a holding file but will have
little inside of it. The son will be even more tiny since it ran after
the grandfather and father backups.

The first Saturday will be at the end of the first week of every month.
If you choose first anyOtherDayOfTheWeek, that might not be until the
2nd week of the month. If you choose a day of the month, that varies
all over the place as to which day-of-the-week it falls on.


#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential
backups.

Which is what you get in the free version of Reflect (incrementals are
absent in the free version). If you run Differentials every day, they
are effectively incrementals but based off the full backup. Don't know
how long or at what interval is "periodically". The longer the chain of
differentials, the more fragile it becomes. Should any differential
backup file become corrupt or go missing, all differentials from that
corrupted/missing differential will be unusable.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental
backups.

The problem there is the chain will get longer. You don't want to keep
running full backups at short intervals because of their size. You will
chew up your storage space rather quickly with lots of full backups. If
you run a full backup once per month, that means the chain of 30
incrementals will have 30 links in the chain (plus the full backup). If
you run them over a week, the chain will be shorter, so less loss if the
chain breaks, but you'll be running 4 times as many full backups with
them running every weekend. Whether that is doable depends on how large
is the backup storage media and how big is the source that is getting
backed up.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever
creating a single Full backup. After this Incremental backups are
created ad infinitum. The Full backup is consolidated with subsequent
Incremental backups once the specified number of Incremental backups
is reached. This is also known as a Synthetic Full backup.

That does one full backup followed thereafter with incrementals. The
chain keeps getting longer and longer. However, if you shorten the
retention interval, the incrementals that are older than that will get
rolled into the full backup (which means you are touching the full
backup, the primary one, instead of leaving it alone for use only during
a restore). Retouching the base (full) backup repeatedly is a recipe
for disaster. The idea is to keep the chain shorter by limiting how
many incrementals to reduce the chain length, but that is at the risk of
touching the full backup.

The other problem is that restoring the full backup (so you can then use
the incrementals in a restore) is that you will be restoring EVERYTHING
ever changed on your computer. Something you installed but later
uninstalled, some registry tweak you changed but unchanged, and anything
changed and unchanged will be recorded in both states in the oversized
full backup file. Rolling in incrementals doesn't eliminate any entries
in the full backup that would be voided by later changes in the
incrementals. The full backup that keeps getting changed (instead of
keeping it safely quiescent) will keep getting larger (unless you make
no changes at all).

Oh, and how often do others make a new recovery USB?

When there is a major version change to Reflect. The PE doesn't change
what it uses (since you keep using the same version that is compatible
with the installed version of Windows), but features or fixes will
change in a new version which could apply to using Reflect under the PE.

For the boot-time image (to load Reflect on booting the computer which
means storing the boot image on an internal drive), and when there is a
sub-minor version change (e.g., 7.1 to 7.2), I remove Reflect's
boot-time option, and reinstall it (which builds a new boot image and
stores on the internal drive). That is pretty easy. Just go into the
menu, remove the boot option, re-enable the boot option.

The Windows PE under which Reflect will run when booted won't change.
You still use the same WinPE as before that is compatible with your
installed version of Windows.

For the bootable optical disc (CD), I don't bother burning a new one
until there is a major change to Reflect (e.g., version 6.x to 7.x).
When I restore, I'm highly likely to use the boot-time image: boot the
computer, choose with OS image to load, very easy. The CD is a backup
should the drive get corrupted, fried, died, or replaced where are the
boot images.

Which bring up another point: If the storage media goes bad where you
saved your backup files, how are you going to restore? If the drive
goes bad or is lost, you no longer have any of your backups. I have an
internal HDD (be a waste to use an SDD) to which I save my backups. You
did not say which *edition* of Reflect that you have. I have the Home
edition, and that does not have the option to copy the finished backup
to another location, like to another drive (which could be a USB drive).
It doesn't even have an option to copy the backup file via FTP to
another host. I'd have to pay more to get the auto-copy and FTP
options.

Instead I used to use the free SyncBack program, but have since moved up
to the SyncBack Lite version (because it could copy inuse/locked files
using the Volume Shadow Copy feature in Windows). That wasn't needed
for Reflect, but for other file-sync jobs that I wanted SyncBack to do.
The Lite version has since been retired, and they replaced it with
SyncBack SE (which, as I recall, is $10 more than Lite's price). I
never bothered to leave the Lite version, because they still support it
with updates, so I don't know if the $10 gives me anything more than
what the Lite version gives me for features (and those I'd use). I use
SyncBack (even free version works) to copy the backup files from the
internal SATA HDD to a USB HDD. That gives me a copy of the backup
files. If the USB HDD fails, I get another. If the internal SATA HDD
fails, use the USB HDD in the interim for any restores, and replace the
internal SATA HDD to copy the backups from the USB HDD (since restores
are much faster using the internal SATA HDD, and the USB HDD is really
just a backup of my backups).

After Reflects ends a backup job, the backup file won't be inuse or
locked, so even the free version of SyncBack will work to copy my
backups files from the internal HDD to the USB HDD. Alas, Reflect
doesn't have easy-to-configure pre- and post-backup job definitions. It
has no means of defining to run a script (batch file) before or after
its backup. It does let you define, for example, MS-DOS batch files
that will run the backup, and you can edit them to add commands before
and after the command to run the backup. Alternatively, Reflect uses
the Windows Task Scheduler to define its scheduled backups, so you could
copy the command line for its scheduled event, put it in a batch file,
edit it to add pre- and post-backup commands, and create a new scheduled
event in Task Scheduler (and disable/delete the one that Refect
created). I've run into problems doing either method which has both use
a .bat script that can be edited when used on a Windows 10 host. When
the batch files run, Reflect knows how to elevate itself to admin
privileges, but other commands you run in the batch file won't be
elevated which means they may fail. Yes, you can edit the Task
Scheduler event to run with admin privileges (escalated), but that won't
work if they run under a Microsoft account instead of a local/offline
account (where both have admin privileges). There are reasons why I
want the backups to run under my account, and reasons why I still use a
Microsoft account instead of a local/offline account.

Task Scheduler in Windows 10 only knows how to handle admin-level
scheduled events that use a local/offline account. For Microsoft
accounts, Task Scheduler doesn't know how to access the login
credentials for that account. That means you cannot run events as
admin, because Task Scheduler requires it knows the login credentials
for an admin account, but Task Scheduler cannot get the login
credentials for a Microsoft account. Yeah, Microsoft ****ed up in Task
Scheduler to run events under a Microsoft account even when it has admin
privileges. There are few (not many, but a couple) advantages to using
a Microsoft account. I could run the events in Task Scheduler under a
different account, but that has problems (too long to get into this
already very long reply). I found a trick of defining an event in Task
Scheduler that triggers on an event, and when Reflect finishes a backup
an event gets recorded (you can see it in the Event Viewer). I first
have SyncBack create its scheduled sync jobs in Task Scheduler. In Task
Scheduler, I edit those events to change the run trigger from specifying
a time to run "On an event". The "scheduled" events for SyncBack would
fire when the Reflect backups finished. The definition of the event
trigger is difficult to understand, and you have to read the event to
get the ID, and the custom event script is hard to decipher, so it's
tricky and I won't get into it here. If you're not using a Microsoft
account under which you run your Reflect backups (you're using a local
aka offline account with admin privs), you can use Reflect's MS-DOS
Batch Files tab to let you edit .bat files to add commands before and
after the backup starts/ends, like running SyncBack after the backup to
save a copy of the backup file to another drive or even another host
(SyncBack will do file sync to local drives whether internal or USB
attached, to network mapped drives, or to other hosts via FTP).

As for the USB drive letters changing, not a problem when the backup
files get saved to an internal HDD where drive don't change unless you
commit surgery on your disk setup. Reflect Home doesn't have an option
to target a USB drive other than by a drive letter. SyncBack (well, in
the Lite version, don't have the free version to check) lets you specify
the target drive by drive letter, serial number, or label. For example,
my USB-attached HDD is drive E:, serial 6E89-9DA2, or label WDC2TB_USBHD
(for a Western Digital 2TB USB HD). I give my each of USB drives a
unique label, so I could use that to find them no matter which drive
letter they happen to get. Using the serial number would get confusing
to figure out later which I had specified a long time ago. Even if the
drive letter changed, SyncBack can still find the USB HDD. A feature
available in SyncBack but not in Reflect.

Alternatively, there are tools that will try to keep the drive letter
assignment the same between ejects and injects of a USB drive, like
UWE-Sieber's USBDLM (USB Drive Letter Manager) at:

https://www.uwe-sieber.de/usbdlm_e.html

Obviously if, say, drive letter F: has already been assigned to a drive
then plugging in the USB drive won't let it get F: again. You have to
use USBDLM to define which drive letter gets assigned to which USB
device for all of them you may use, so each gets a unique one to
eliminate the conflict in the first place. Lots of prep work, plus you
may discard and get new USB drives and have to update USBDLM again.
Normally Windows should not change the drive letter of a USB drive
unless the old drive letter it had before is currently in use. I do
remember having to sometimes go into Disk Management (diskmgmt.msc) to
assign a drive letter there for it to get remembered then the USB drive
got plugged in later. I don't remember the circumstances, but assigning
the drive letter in Disk Management was better at making sticky the
drive letter assignment (as long as there was no conflict in drive
letter assignments).

Still awake?

VanguardLH wrote:

Terry Pinnell wrote:

Some background before I get to my specific question.

I setup Macrium Reflect (Free) over three years ago after helpful
advice here. After a few months of occasional checks I left it to do
its stuff and essentially forgot about it. It was scheduled very
early every Saturday morning so I didn't often get to see it start
anyway. But yesterday I discovered that every weekly run since May
had been unsuccessful ;-(

(The reason turned out to be that somehow the drive letter of my
destination external USB drive letters had been altered.)

--------------------

In the course of reconfiguring I'm therefore reconsidering my choice
of various scheduling templates, I think my previous was
'Incremental Backup Set', and that now seems perhaps a bit over
complicated for a solo end user with no network stuff? I'd appreciate
others' opinions or experience please.

METHODS
#1 Grandfather, Father, Son:
Daily Incremental ("Son"), weekly Differential ("Father"), and monthly
Full ("Grandfather") backups.

Reflect doesn't do true GFS backups where, for example, a grandfather
father and son scheduled at the same time would only have the
grandfather backup run. I haven't found a consumer-grade backup program
that does a true GFS. Instead they just use the GFS concept for
scheduling. If you schedule them at the same time, you don't which will
run. If you schedule the grandfather with the father starting 5 minutes
later and the son starting 10 minutes later, you know the order they
will run: grandfather, father, and then son. Reflect will only run one
backup at a time, so the father pends until the grandfather finishes,
and the son pends until the father finishes.

That I did was to schedule the grandfather backup (full) to run on the
first Saturday of the month, the father backup (differential) on Sunday
(at the same time but different day as the grandfather), and the son
backup (incremental) to start 5 minutes later on every day. I could
have grandfather run on the first Saturday, father on the first Saturday
but 5 minutes later, and the son on every day 10 minutes later, too.
Since the grandfather just ran, the father would be very small. Not
much changed between when the grandfather and father backups ran, and
the father backup is mostly the size of a holding file but will have
little inside of it. The son will be even more tiny since it ran after
the grandfather and father backups.

The first Saturday will be at the end of the first week of every month.
If you choose first anyOtherDayOfTheWeek, that might not be until the
2nd week of the month. If you choose a day of the month, that varies
all over the place as to which day-of-the-week it falls on.


#2 Differential Backup Set:
A Full backup is created periodically followed by daily Differential
backups.

Which is what you get in the free version of Reflect (incrementals are
absent in the free version). If you run Differentials every day, they
are effectively incrementals but based off the full backup. Don't know
how long or at what interval is "periodically". The longer the chain of
differentials, the more fragile it becomes. Should any differential
backup file become corrupt or go missing, all differentials from that
corrupted/missing differential will be unusable.

#3 Incremental Backup Set:
A Full backup is created periodically followed by daily Incremental
backups.

The problem there is the chain will get longer. You don't want to keep
running full backups at short intervals because of their size. You will
chew up your storage space rather quickly with lots of full backups. If
you run a full backup once per month, that means the chain of 30
incrementals will have 30 links in the chain (plus the full backup). If
you run them over a week, the chain will be shorter, so less loss if the
chain breaks, but you'll be running 4 times as many full backups with
them running every weekend. Whether that is doable depends on how large
is the backup storage media and how big is the source that is getting
backed up.

#4 Incrementals Forever:
Incrementals forever optimizes backup space and time by only ever
creating a single Full backup. After this Incremental backups are
created ad infinitum. The Full backup is consolidated with subsequent
Incremental backups once the specified number of Incremental backups
is reached. This is also known as a Synthetic Full backup.

That does one full backup followed thereafter with incrementals. The
chain keeps getting longer and longer. However, if you shorten the
retention interval, the incrementals that are older than that will get
rolled into the full backup (which means you are touching the full
backup, the primary one, instead of leaving it alone for use only during
a restore). Retouching the base (full) backup repeatedly is a recipe
for disaster. The idea is to keep the chain shorter by limiting how
many incrementals to reduce the chain length, but that is at the risk of
touching the full backup.

The other problem is that restoring the full backup (so you can then use
the incrementals in a restore) is that you will be restoring EVERYTHING
ever changed on your computer. Something you installed but later
uninstalled, some registry tweak you changed but unchanged, and anything
changed and unchanged will be recorded in both states in the oversized
full backup file. Rolling in incrementals doesn't eliminate any entries
in the full backup that would be voided by later changes in the
incrementals. The full backup that keeps getting changed (instead of
keeping it safely quiescent) will keep getting larger (unless you make
no changes at all).

Oh, and how often do others make a new recovery USB?

When there is a major version change to Reflect. The PE doesn't change
what it uses (since you keep using the same version that is compatible
with the installed version of Windows), but features or fixes will
change in a new version which could apply to using Reflect under the PE.

For the boot-time image (to load Reflect on booting the computer which
means storing the boot image on an internal drive), and when there is a
sub-minor version change (e.g., 7.1 to 7.2), I remove Reflect's
boot-time option, and reinstall it (which builds a new boot image and
stores on the internal drive). That is pretty easy. Just go into the
menu, remove the boot option, re-enable the boot option.

The Windows PE under which Reflect will run when booted won't change.
You still use the same WinPE as before that is compatible with your
installed version of Windows.

For the bootable optical disc (CD), I don't bother burning a new one
until there is a major change to Reflect (e.g., version 6.x to 7.x).
When I restore, I'm highly likely to use the boot-time image: boot the
computer, choose with OS image to load, very easy. The CD is a backup
should the drive get corrupted, fried, died, or replaced where are the
boot images.

Which bring up another point: If the storage media goes bad where you
saved your backup files, how are you going to restore? If the drive
goes bad or is lost, you no longer have any of your backups. I have an
internal HDD (be a waste to use an SDD) to which I save my backups. You
did not say which *edition* of Reflect that you have. I have the Home
edition, and that does not have the option to copy the finished backup
to another location, like to another drive (which could be a USB drive).
It doesn't even have an option to copy the backup file via FTP to
another host. I'd have to pay more to get the auto-copy and FTP
options.

Instead I used to use the free SyncBack program, but have since moved up
to the SyncBack Lite version (because it could copy inuse/locked files
using the Volume Shadow Copy feature in Windows). That wasn't needed
for Reflect, but for other file-sync jobs that I wanted SyncBack to do.
The Lite version has since been retired, and they replaced it with
SyncBack SE (which, as I recall, is $10 more than Lite's price). I
never bothered to leave the Lite version, because they still support it
with updates, so I don't know if the $10 gives me anything more than
what the Lite version gives me for features (and those I'd use). I use
SyncBack (even free version works) to copy the backup files from the
internal SATA HDD to a USB HDD. That gives me a copy of the backup
files. If the USB HDD fails, I get another. If the internal SATA HDD
fails, use the USB HDD in the interim for any restores, and replace the
internal SATA HDD to copy the backups from the USB HDD (since restores
are much faster using the internal SATA HDD, and the USB HDD is really
just a backup of my backups).

After Reflects ends a backup job, the backup file won't be inuse or
locked, so even the free version of SyncBack will work to copy my
backups files from the internal HDD to the USB HDD. Alas, Reflect
doesn't have easy-to-configure pre- and post-backup job definitions. It
has no means of defining to run a script (batch file) before or after
its backup. It does let you define, for example, MS-DOS batch files
that will run the backup, and you can edit them to add commands before
and after the command to run the backup. Alternatively, Reflect uses
the Windows Task Scheduler to define its scheduled backups, so you could
copy the command line for its scheduled event, put it in a batch file,
edit it to add pre- and post-backup commands, and create a new scheduled
event in Task Scheduler (and disable/delete the one that Refect
created). I've run into problems doing either method which has both use
a .bat script that can be edited when used on a Windows 10 host. When
the batch files run, Reflect knows how to elevate itself to admin
privileges, but other commands you run in the batch file won't be
elevated which means they may fail. Yes, you can edit the Task
Scheduler event to run with admin privileges (escalated), but that won't
work if they run under a Microsoft account instead of a local/offline
account (where both have admin privileges). There are reasons why I
want the backups to run under my account, and reasons why I still use a
Microsoft account instead of a local/offline account.

Task Scheduler in Windows 10 only knows how to handle admin-level
scheduled events that use a local/offline account. For Microsoft
accounts, Task Scheduler doesn't know how to access the login
credentials for that account. That means you cannot run events as
admin, because Task Scheduler requires it knows the login credentials
for an admin account, but Task Scheduler cannot get the login
credentials for a Microsoft account. Yeah, Microsoft ****ed up in Task
Scheduler to run events under a Microsoft account even when it has admin
privileges. There are few (not many, but a couple) advantages to using
a Microsoft account. I could run the events in Task Scheduler under a
different account, but that has problems (too long to get into this
already very long reply). I found a trick of defining an event in Task
Scheduler that triggers on an event, and when Reflect finishes a backup
an event gets recorded (you can see it in the Event Viewer). I first
have SyncBack create its scheduled sync jobs in Task Scheduler. In Task
Scheduler, I edit those events to change the run trigger from specifying
a time to run "On an event". The "scheduled" events for SyncBack would
fire when the Reflect backups finished. The definition of the event
trigger is difficult to understand, and you have to read the event to
get the ID, and the custom event script is hard to decipher, so it's
tricky and I won't get into it here. If you're not using a Microsoft
account under which you run your Reflect backups (you're using a local
aka offline account with admin privs), you can use Reflect's MS-DOS
Batch Files tab to let you edit .bat files to add commands before and
after the backup starts/ends, like running SyncBack after the backup to
save a copy of the backup file to another drive or even another host
(SyncBack will do file sync to local drives whether internal or USB
attached, to network mapped drives, or to other hosts via FTP).

As for the USB drive letters changing, not a problem when the backup
files get saved to an internal HDD where drive don't change unless you
commit surgery on your disk setup. Reflect Home doesn't have an option
to target a USB drive other than by a drive letter. SyncBack (well, in
the Lite version, don't have the free version to check) lets you specify
the target drive by drive letter, serial number, or label. For example,
my USB-attached HDD is drive E:, serial 6E89-9DA2, or label WDC2TB_USBHD
(for a Western Digital 2TB USB HD). I give my each of USB drives a
unique label, so I could use that to find them no matter which drive
letter they happen to get. Using the serial number would get confusing
to figure out later which I had specified a long time ago. Even if the
drive letter changed, SyncBack can still find the USB HDD. A feature
available in SyncBack but not in Reflect.

Alternatively, there are tools that will try to keep the drive letter
assignment the same between ejects and injects of a USB drive, like
UWE-Sieber's USBDLM (USB Drive Letter Manager) at:

https://www.uwe-sieber.de/usbdlm_e.html

Obviously if, say, drive letter F: has already been assigned to a drive
then plugging in the USB drive won't let it get F: again. You have to
use USBDLM to define which drive letter gets assigned to which USB
device for all of them you may use, so each gets a unique one to
eliminate the conflict in the first place. Lots of prep work, plus you
may discard and get new USB drives and have to update USBDLM again.
Normally Windows should not change the drive letter of a USB drive
unless the old drive letter it had before is currently in use. I do
remember having to sometimes go into Disk Management (diskmgmt.msc) to
assign a drive letter there for it to get remembered then the USB drive
got plugged in later. I don't remember the circumstances, but assigning
the drive letter in Disk Management was better at making sticky the
drive letter assignment (as long as there was no conflict in drive
letter assignments).

Still awake?

Many thanks.

Yes, still awake, but will re-read more carefully tomorrow!

This is Macrium Reflect 7.2 (Free), 64-bit.

Terry, East Grinstead, UK

Terry Pinnell wrote:

Wow, that was an impressively thorough explanation. Collectively with Paul's this
thread is now becomes 'Macrium Scheduling Tutorial'!

I was confused as to which of those four 'methods' I listed in my opening post I've
currently specified. Instead of a link I was trying to be helpful in pasting, but
should have avoided adding my own (wrong) numbers!

The XML indicates that I'm using 'method 3', which I now see is the fourth option,
not the third. And I gather 'Auto' is 'Incrementals Forever' or 'Synthetic Full
Backup'.

Here's my current schedule, specified yesterday:

https://www.dropbox.com/s/7mawbl3rvc...le-1.jpg?raw=1

Bottom line: I reckon that with simplicity as a key motivator for me, and no
capacity issue, I'm favouring just Fulls, weekly.

Terry, East Grinstead, UK

As long as the ratio of a Full, to the size of the backup
device, leaves room for lots of backups, then your situation
will be "mostly maintenance free". You'll only have cleanup or
migration to do "once in a while".

It's when there is a danger of running out of space, that
more cunning is required.

I would likely have to buy a larger external for myself,
if three fulls couldn't fit on the external drive. That would
be my tolerance for pain. Any amount of space less than that,
I'd just do them manually, with no schedule. (Which is
what I do here anyway.)

When my Internet broke about a week ago, I passed the time
by doing a Full :-) That's my "schedule".

The only drive that would make a decent external for me,
would be the new 100TB SSD they came out with. (No price
was provided, and it just looked like a mockup, rather than
the real thing. Form factor was 3.5".) That should be
more expensive than a small car. Then, just about any
schedule I used, would work.

Paul

Terry Pinnell wrote:

Bottom line: I reckon that with simplicity as a key motivator for me, and no
capacity issue, I'm favouring just Fulls, weekly.

I like to do a scheduled backup every day in the early morning (while
I'm sleeping). I don't know when Microsoft might push yet another
update. Some can be delayed, like the 1909 update, but many get shoved
onto your computer awaiting the next reboot. With Microsoft's history
of pushing bad updates (some causing castastrophies), I want a backup
the morning they push an update, so I have a decent escape route
(Microsoft's system restore or update backouts are not reliable).

I might also flub something in Windows or some program, and need to
restore. I don't want to lose everything for a whole week. I might've
just completed some legal doc that I'm still gathering info to complete,
and need to print and send later. Quite often finding the correct
doc(s) the gov't wants is a big effort. For some, that's why they hire
lawyers to do all that. The gov't loves paperwork, but it has to just
the right paperwork.

The granularity of your backups depends on how much you're willing to
lose should you have to restore from malware, bad updates, a bad tweak,
software installs that are nasty to remove, or failed hardware. To me,
a week is way to long. For others, they may go with a monthly backup.
You could keep using the weekly backups, and watch how many restores you
do that result in too much or critical data or config loss. I started
them monthly. Bang, had to restore, way too much lost. Moved to
weeklies. Bang, had to restore, too much lost. Moved to daily. Since
they are scheduled, it's not like I get nuisanced to run them. I just
have to monitor my backup drives to make sure they're not running out of
room for new backups (which means I've had to tailor the retentions to
automatically free up space).

Paul wrote:

Terry Pinnell wrote:

Wow, that was an impressively thorough explanation. Collectively with Paul's this
thread is now becomes 'Macrium Scheduling Tutorial'!

I was confused as to which of those four 'methods' I listed in my opening post I've
currently specified. Instead of a link I was trying to be helpful in pasting, but
should have avoided adding my own (wrong) numbers!

The XML indicates that I'm using 'method 3', which I now see is the fourth option,
not the third. And I gather 'Auto' is 'Incrementals Forever' or 'Synthetic Full
Backup'.

Here's my current schedule, specified yesterday:

https://www.dropbox.com/s/7mawbl3rvc...le-1.jpg?raw=1

Bottom line: I reckon that with simplicity as a key motivator for me, and no
capacity issue, I'm favouring just Fulls, weekly.

Terry, East Grinstead, UK

As long as the ratio of a Full, to the size of the backup
device, leaves room for lots of backups, then your situation
will be "mostly maintenance free". You'll only have cleanup or
migration to do "once in a while".

It's when there is a danger of running out of space, that
more cunning is required.

I would likely have to buy a larger external for myself,
if three fulls couldn't fit on the external drive. That would
be my tolerance for pain. Any amount of space less than that,
I'd just do them manually, with no schedule. (Which is
what I do here anyway.)

When my Internet broke about a week ago, I passed the time
by doing a Full :-) That's my "schedule".

The only drive that would make a decent external for me,
would be the new 100TB SSD they came out with. (No price
was provided, and it just looked like a mockup, rather than
the real thing. Form factor was 3.5".) That should be
more expensive than a small car. Then, just about any
schedule I used, would work.

Paul

Can you say why your images are so gigantic? A couple of others I've communicated
with regard my 180 GB (imaging to about 135 GB) as being very large compared to
their 30-50 GB. Is it because you're imaging almost your entire PC contents? **

When I set up this PC in 2016 I didn't feel technically confident enough about
partitioning to split the OS and the rest on my 256 GB SSD, C. (Still don't really.)
And I felt more comfortable having all my several hundred programs on it as well as
the OS, in the hope that would make for easier recovery. And I have data there too,
mostly those files associated with my video-making which need especially fast
access. (The great bulk of my data is on my 4 TB internal HD, D.) But if I had split
off the OS then I could now simply use the Macrium option "Create an image of the
partitions required to back up and restore Windows", which would be much smaller.


** After taking advantage of the Black Friday 50% discount yesterday I upgraded.
Making my first Full I was dismayed to see an estimated 7 hours for completion and
something like 1.3 TB. (Previous Fulls had taken about 30 mins.) Turned out I had
clicked 'Image selected disks on this computer'; only two of the small boxes against
my 7 drives were checkmarked and I'd hastily assumed that was another neat way of
starting an image. (I'm still not exactly clear how those boxes are meant to be
used.) After promptly canceling and then reverting to the correct method of clicking
'Image this disk' below C in the main window, it made the image (on my 3 TB external
USB drive I) in about 30 mins.

BTW, I'm backing up all non-OS files nightly and weekly, which is why I'm confining
imaging to C.

Terry, East Grinstead, UK

VanguardLH wrote:

Terry Pinnell wrote:

Bottom line: I reckon that with simplicity as a key motivator for me, and no
capacity issue, I'm favouring just Fulls, weekly.

I like to do a scheduled backup every day in the early morning (while
I'm sleeping). I don't know when Microsoft might push yet another
update. Some can be delayed, like the 1909 update, but many get shoved
onto your computer awaiting the next reboot. With Microsoft's history
of pushing bad updates (some causing castastrophies), I want a backup
the morning they push an update, so I have a decent escape route
(Microsoft's system restore or update backouts are not reliable).

Indeed. OT in this thread but... I may post again about my frequently vanishing SRs.
I might also flub something in Windows or some program, and need to
restore. I don't want to lose everything for a whole week. I might've
just completed some legal doc that I'm still gathering info to complete,
and need to print and send later. Quite often finding the correct
doc(s) the gov't wants is a big effort. For some, that's why they hire
lawyers to do all that. The gov't loves paperwork, but it has to [be] just
the right paperwork.

That's another chord that rings loud for me. OT again but... I have some stock from
my employee share plan with IBM UK. For reasons I've never fathomed it's managed in
the USA, not UK. I've just spent several *weeks* (translantic calls, emails and
surface mail) meeting the IRS's need for a form called W-8BEN, completed to rigid
perfection.

The granularity of your backups depends on how much you're willing to
lose should you have to restore from malware, bad updates, a bad tweak,
software installs that are nasty to remove, or failed hardware. To me,
a week is way to long. For others, they may go with a monthly backup.
You could keep using the weekly backups, and watch how many restores you
do that result in too much or critical data or config loss. I started
them monthly. Bang, had to restore, way too much lost. Moved to
weeklies. Bang, had to restore, too much lost. Moved to daily. Since
they are scheduled, it's not like I get nuisanced to run them. I just
have to monitor my backup drives to make sure they're not running out of
room for new backups (which means I've had to tailor the retentions to
automatically free up space).

You're running G-F-S, yes? How much capacity have you allocated for your images in
total?

Although ideally I too would prefer daily imaging, my leaning towards weekly is that
my nightly and weekly backups (using SecondCopy for decades) would allow speedy
recovery of data. Hardware disasters apart, of course. Anyway, I'll see how it goes
and maybe change my mind.

Terry, East Grinstead, UK

Terry Pinnell wrote:
Paul wrote:

Terry Pinnell wrote:

Wow, that was an impressively thorough explanation. Collectively with Paul's this
thread is now becomes 'Macrium Scheduling Tutorial'!

I was confused as to which of those four 'methods' I listed in my opening post I've
currently specified. Instead of a link I was trying to be helpful in pasting, but
should have avoided adding my own (wrong) numbers!

The XML indicates that I'm using 'method 3', which I now see is the fourth option,
not the third. And I gather 'Auto' is 'Incrementals Forever' or 'Synthetic Full
Backup'.

Here's my current schedule, specified yesterday:

https://www.dropbox.com/s/7mawbl3rvc...le-1.jpg?raw=1

Bottom line: I reckon that with simplicity as a key motivator for me, and no
capacity issue, I'm favouring just Fulls, weekly.

Terry, East Grinstead, UK
As long as the ratio of a Full, to the size of the backup
device, leaves room for lots of backups, then your situation
will be "mostly maintenance free". You'll only have cleanup or
migration to do "once in a while".

It's when there is a danger of running out of space, that
more cunning is required.

I would likely have to buy a larger external for myself,
if three fulls couldn't fit on the external drive. That would
be my tolerance for pain. Any amount of space less than that,
I'd just do them manually, with no schedule. (Which is
what I do here anyway.)

When my Internet broke about a week ago, I passed the time
by doing a Full :-) That's my "schedule".

The only drive that would make a decent external for me,
would be the new 100TB SSD they came out with. (No price
was provided, and it just looked like a mockup, rather than
the real thing. Form factor was 3.5".) That should be
more expensive than a small car. Then, just about any
schedule I used, would work.

Paul

Can you say why your images are so gigantic? A couple of others I've communicated
with regard my 180 GB (imaging to about 135 GB) as being very large compared to
their 30-50 GB. Is it because you're imaging almost your entire PC contents? **

When I set up this PC in 2016 I didn't feel technically confident enough about
partitioning to split the OS and the rest on my 256 GB SSD, C. (Still don't really.)
And I felt more comfortable having all my several hundred programs on it as well as
the OS, in the hope that would make for easier recovery. And I have data there too,
mostly those files associated with my video-making which need especially fast
access. (The great bulk of my data is on my 4 TB internal HD, D.) But if I had split
off the OS then I could now simply use the Macrium option "Create an image of the
partitions required to back up and restore Windows", which would be much smaller.


** After taking advantage of the Black Friday 50% discount yesterday I upgraded.
Making my first Full I was dismayed to see an estimated 7 hours for completion and
something like 1.3 TB. (Previous Fulls had taken about 30 mins.) Turned out I had
clicked 'Image selected disks on this computer'; only two of the small boxes against
my 7 drives were checkmarked and I'd hastily assumed that was another neat way of
starting an image. (I'm still not exactly clear how those boxes are meant to be
used.) After promptly canceling and then reverting to the correct method of clicking
'Image this disk' below C in the main window, it made the image (on my 3 TB external
USB drive I) in about 30 mins.

BTW, I'm backing up all non-OS files nightly and weekly, which is why I'm confining
imaging to C.

Terry, East Grinstead, UK


I discovered there are controls on the left of the Macrium
window, to back up the entire PC. Whereas I usually work on
one drive at a time, on the right.

"Image selected disks on this computer" === whole computer backup 2.13TB of files

I have a number of VM images, and those add up after a while.
Those are "OSes", stored on data partitions, and used in
things like VPC2007 (WinXP only) or VirtualBox ("everywhere").
I have VMs even sitting on temporary disks.

It's because I have no inventory system, that I can't clean
this stuff up.

I do actually have an Inventory folder, but it's out of date.
It's time for another scan for that thing. It's an all day
job to do that properly.

*******

The speed of a Macrium backup, is constrained by a couple of things.
One is, if you select lightweight compression, I think it only
uses one core for that. Whereas PIGZ or 7ZIP know how to use
multiple cores, and 7ZIP now even knows how to use multiple
cores during an unzip operation. Macrium doesn't seem to be
all that curious about the options available to them.

In addition, Macrium computes an MD5 or SHA1 checksum for
the image it creates. When you do a "Verify" on a .mrimg file
later, that's what it checks (the checksum), to detect whether
the file has been damaged since it was made. And I did detect
a couple of bad images using Verify, so it works.

Macrium cannot back up faster than it can compute (whatever
it uses for the checksum. Even with ideal storage devices,
depending on your CPU it probably can't break about 300MB/sec.
So even if you had two NVMe drives, the backup flowing from
one to the other, would have a ballpark 300MB/sec limit.

The fastest checksum calc I've seen here, is a CRC32 done
at 1500MB/sec or so (7ZIP, menu). Which still "holds back" a good quality
NVMe from showing its stuff. Even if Macrium switched to CRC32
for a checksum (a bad idea!), there would still be transfer
speed limits.

*******

On external drives, make sure the USB enclosure and the USB port,
are real actual USB3 items. Blue colored connectors and so on.
Bench the drive with HDTune and verify the read transfer
rate seems reasonable for the enclosure type.

I find sometimes, that the Windows disk caches misbehave a bit,
and that results in some weird transfer rate fluctuations. When
I see this, it makes me really angry :-) We pay good money for
these OSes, to have this crap fixed and working properly.

Paul

Terry Pinnell wrote:

You're running G-F-S, yes? How much capacity have you allocated for your images in
total?

Reflect images my C: drive (OS + apps + data). I move data to drive D:
that I don't want in the backups occupying disk space, because it is
replaceable or a copy stored elsewhere. C: is a 1TB NVMe SSD, but only
103GB is currently used.

My GFS is monthly full, weekly differential, and daily incrementals.
Retention is: 26 weeks for fulls (monthly), 5 weeks for differentials
(weekly), and 14 days for incrementals (dailies).

The internal D: drive 2TB HDD stores the backup images, program exported
configs/backups, and some rather large data files that don't need to be
in the backups (and why they're on the same drive as the backups which
you wouldn't do if they were in the backups). For example, a couple
versions of WSUSoffline are on D: to save all the MS updates for Win7,
Office 2016/2019, and Win10.

C: SSD drive: 103GB used (of 1TB)
D: HDD drive: 546GB used (of 2TB)
410GB for Reflect image backups (*)
13GB for wsusoffline (only have 1 copy now)
100GB for some large files
E: USB drive: 619GB used (of 2TB)
410GB for copies of Reflect image backups
100GB for copies of some of the large files
9GB for a copy of C:\Downloads (**)

(*) Includes a 70GB folder for Win7 full image taken before wiping and
a fresh install of Win10, just in case I need data from that old setup.
(**) Not the default Downloads folder in Win10 (which is under my
account profile folder and used only for temporary downloads).
C:\DOWNLOADS is where I store all my downloads for all programs, copied
to E: (USB) and occasionally copied to a 64GB USB flash drive.

For my Win10 desktop, image backups only go back to Aug 2019, because
that's when I built the new box. 6 months retention means the old ones
will start rolling out around Jan-Feb 2020. For now, I should have
plenty of free space to accomodate the later backups (along with the
other files I'm saving on the "backup" drives). If I went to a year's
retention, I'd be looking to replace the 2TB drives with 4TB drives, or
larger, depending on my budget at the time. Or, if I consumed a lot
more space on the 1TB NVMe SSD, I'd have to get bigger backup drives.

Note: Not all the image backup files for Reflect are for the scheduled
backup jobs. Occasionally I create an image backup before some change,
like an incremental if I do MS updates in the evening (since there
wouldv'e been an incremental that day early in the morning), or a
differential or full depending on the magnitude of the change for which
I want an escape route for recovery. So, besides the scheduled backups,
there are a few manual backups saved on D: (and then sync'ed to E: for a
2nd copy of the backups).

My old boxes go to family to upgrade their hardware. So, as yet, I
haven't had a 2nd box at home to use to save backup files via FTP. That
would be safer for the backup copies than a USB drive (which has to be
connected and powered on all the time to do the backup file sync to the
USB drive after the scheduled backup jobs). Reflect has its Image
Guardian which acts like a rootkit by injecting its driver in the OS
file I/O APIs to protect against writes and deletes of the backup files
(but doesn't prevent reading them).

Image Guardian helps a bit against ransomware for either the backup
files on my D: "backup" internal HDD drive and the backup of backup
files on the E: USB HDD drive. However, if I can figure out how to
workaround Image Guardian, I'm sure malware can, too. I discussed the
window of opportunity for malware to get past Image Guardian with
Macrium, they agreed, but they weren't interested in a more thorough
protection scheme. They added something to bolster their backup
software against most ransomware, and it suffices for that.

Anything that is currently mounted can be found by ransomware or
malware. Hell, all it has to do is run diskpart and run "list volume".
Drive letters are not required to get at partitions. Acronis, Paragon,
and another (I forget which) can store backups in "hidden" partitions:
the partition type number is non-standard, so standard disk tools should
leave it alone, and no drive letter is assigned (which is the "hiding"
part), but those backup programs can still access the partitions to
write and read their backup files from there. Not assigning a drive
letter to a partition doesn't block access to the partition, so it's a
weak anti-ransomware protection scheme, but better than nothing.
Programs can use mount points. Humans have been trained to recognize
drive letters.

With backup files getting copied to an FTP server, malware can't reach
those backups because they won't know the login credentials for the
account under which the backups get uploaded to the FTP host, and other
modes of access would be locked out. For scheduled backup to run and
then send a copy to the FTP server, the login credentials would have to
be stored but be secure to prevent malware from finding the login
credentials. FTPS would be needed to secure the login credentials
during transit to prevent sniffing by malware.