places.sqlite read time out

T wrote:
On 7/6/19 7:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite
Read timed out.
The customer leaves Firefox running 24/7

A backup using Cobian to a local FTP server that normally
takes 20 minutes was taking over 24 hours.

I always close Firefox before backing up my Profile;
I "RoboCopy" backup my entire profile,
and select files are uploaded to my FTP server.


customer leaves his computer running 24/7

:'(

And you have to engineer accordingly.

It's a "tall order" to expect your customer to be
VSS aware - and also a problem to be killing his stuff
at 3AM in the morning when they're not there.

Some kinda no-win situation.

Paul

Jeff-Relf.Me @.@ wrote:
close Firefox before backing up a Profile
customer leaves his computer running 24/7 :'(

Tell the customer that he has to close Firefox.

You hire "IT help" to make things possible.

Not to create extra handcuffs.

With the right setup, you can fix anything.

What Todd is trying to do, is not spend a lot of
money, to make this stuff look like magic.

If you taskkill Firefox, the session information
should already be recorded. Then, you need to
restart Firefox later, fork a copy, somehow pass
an argument that says "yes, please restore
the session". That will make a mess of the
customer desktop of course (windows won't be
properly iconified), but at least there
will be (less) damage due to the backup needs.

The session restore feature is not bulletproof.
It would **** me off, if I lost some windows or
tabs because someone monkeyed with my stuff at 3AM.

*******

http://forums.mozillazine.org/viewto...f=39&t=2987219

"I've never heard of Thunderbird supporting VSS and
would be surprised if that feature was added since
most users have never heard of VSS, and the developers
appear to be reluctant to add platform specific features."

I think that's a fair assessment of the development
environment at Mozilla.

Paul

On 7/6/19 6:55 PM, Paul wrote:
andÂ*alsoÂ*aÂ*problemÂ*toÂ*beÂ*killingÂ*hisÂ*stuff
atÂ*3AMÂ*inÂ*theÂ*morningÂ*whenÂ*they'reÂ*notÂ*the re.

Oh he doesn't care. He just restarts what I kill.
I have remote access to his machine and turn his
stuff off all the time. He can't remember if he
did it or I did it.

On 7/6/19 9:48 PM, Paul wrote:
TellÂ*theÂ*customerÂ*thatÂ*heÂ*hasÂ*toÂ*closeÂ*Fir efox.

YouÂ*hireÂ*"ITÂ*help"Â*toÂ*makeÂ*thingsÂ*possible.


1+

A lot of time, I get advice back that is geared to an
advanced user, not a customer.

T wrote:

I am mainly concerned about the saved password and account.

Do they use Firefox Sync?

On 7/7/19 1:15 AM, VanguardLH wrote:
T wrote:

I am mainly concerned about the saved password and account.

Do they use Firefox Sync?


I am the only one I know that does.

Paul wrote:

VSS manages a certain phase of the backup.

1) Request comes into VSS - "Make a shadow of C: please"
2) VSS sends command to running applications to quiesce.
3) Waits ten seconds for compliance.

Given that firefox/sqlite is not VSS aware, will VSS even bother asking
it to quiesce? Presumable apps have to declare themselves snapshot capable?

On 7/6/2019 1:36 PM, Frank Slootweg wrote:
Out of interest: Do you shutdown Thunderbird during backup? And what
about any other programs which use some kind of (pseudo) database behind
the scenes?
I always close Thunderbird also, as it appears to work like Firefox.

With the volumn of email in my folders, I can not afford not to close
the program before backing up.

--
Judge your ancestors by how well they met their standards not yours.
They did not know your standards, so could not try to meet them.

VanguardLH wrote:
Frank Slootweg wrote:

FWIW, I use a program (NetWorx [1]) which also uses a sqlite database.
I also use Cobian Backup, do *not* shutdown the program/database and
I do *not* get an error.

[1] https://www.softperfect.com/products/networx

Are you sure Networx does not place any database records, extends, or
delta updates in memory (which is not covered in a file/disk backup)?

https://www.sqlite.org/inmemorydb.html

No, I'm not *sure*, but I do not think so. Networx' sqlite database is
kept in a file, but I cannot rule out that that file is used to fill an
sqlite 'In-Memory Database', which is later written back to the disk
file.

The modification time of the database file is currently 16:50 on July
6, i.e. yesterday. That might imply that the active database is not in
that file, but I don't think Windows updates the modification time of
open files (i.e. only at close or specific request).

N.B. Besides the database file (NetWorx.db), there are two other files
(NetWorx.db-wal and NetWorx.db-shm), where the .db-wal file has nearly
the same timestamp (16:51), but the .db-shm (shared memory?) file is of
June 13 at 11:15, which is the time of the last boot.

Anyway, my example was just one of similar circumstances as T's
scenario (sqlite database being backed up by Cobian Backup), which did
not give an error (from Cobian Backup).

Not sure how you could include a drive in a backup job definition that
is a RAM disk, especially one that is created on-the-fly during
operation, and may contain only some parts of the database that would
benefit most from the speed of RAM instead of being on disk.

https://www.softperfect.com/contact/...php?article=44

AFAICT, sqlite 'In-Memory Databases' are not using a RAM disk, just
normal memory, so I don't see how your point is related to the
scenario(s) at hand.

That said, my compliments for your - as always - very thorough
research!

Keith Nuttle wrote:
On 7/6/2019 1:36 PM, Frank Slootweg wrote:
Out of interest: Do you shutdown Thunderbird during backup? And what
about any other programs which use some kind of (pseudo) database behind
the scenes?

I always close Thunderbird also, as it appears to work like Firefox.

With the volumn of email in my folders, I can not afford not to close
the program before backing up.

Fair enough. For existing email (i.e. already received), Thunderbird
should be able to repair an e-mail folder in case its folder index
(.msf) file becomes damaged (right-click folder - Properties - Repair
Folder). But I don't know what could happen (on restore) if a backup
starts while reciving or sending a message, so your precaution is (of
course :-)) a wise one.

Having said that, in 16 years of Outlook Express - Windows Mail -
Thunderbird, I have lost only one [1] message (in Thunderbird! :-)) due
some folder corruption issue.

[1] That I'm *aware* of! :-)

Andy Burns wrote:
Paul wrote:

VSS manages a certain phase of the backup.

1) Request comes into VSS - "Make a shadow of C: please"
2) VSS sends command to running applications to quiesce.
3) Waits ten seconds for compliance.

Given that firefox/sqlite is not VSS aware, will VSS even bother asking
it to quiesce? Presumable apps have to declare themselves snapshot
capable?

VSS probably would not bother.

https://docs.microsoft.com/en-us/win.../the-vss-model

"Consistent file state via application coordination.

VSS provides a COM-based, event-driven interprocess communication mechanism
that participating processes can use to determine system state with respect
to backup, restore, and shadow copy (volume capture) operations.

These events define stages by which applications modifying data on disk
(writers) can bring all their files into a consistent state prior to the
creation of the shadow copy."

If your program keeps a file open, then chances are you'd have
to add support to listen for a request of some sort.

Paul

T wrote:
Hi All,

Windows Nein, oops Windows Ten - build 1809
Firefox 67.0.4, 64 bit

This is from Cobian Backup running Volume Shadow Copy Service (VSS).

ERR 2019-07-02 14:07 Error uploading the file
"GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite":
Read timed out.

Is places.sqlite playing tricks with VSS? Are there things
VSS can't copy?

The customer does leave Firefox running 24/7

Up time was 11 days. Maybe that was the problem. A backup
using Cobian to a local FTP server that normally takes 20
minutes was taking over 24 hours.

Many thanks,
-T

I can recount a "funny" for ya, related to your question.

I have a Win7 setup on an SSD that was just sitting there.

I'm not all that religious about keeping Firefox up to date,
so I needed a Windows install where I could put 67.0.4 and test it
out.

My first test was going to involve Macrium doing a VSS based
backup, while Firefox is running. I figured I should have
at least one news site open, to make the Firefox scenario
"realistic". Like, if the stupid compositing engine that
runs at 60Hz in there, would have "something to do".

I then requested a backup, from the Win7 on SSD, to a hard
drive that happened to be sitting in the computer. The SSD
(being fast), would then be throttled by the HDD destination
for the MRIMG (being slow).

This causes the possibility of write caching (which runs on
Win7 as well as Win10). Write caching uses up system memory,
and in tight memory situations, that can be a problem.

So I go out into the kitchen to make a snack, not really
expecting anything "unique" to be happening. I come back
and the Test Machine is suspiciously quiet. No HDD LED.
The screen is frozen. I fiddle with the mouse a bit, do the
"shift key test", and eventually get a response.
Now, I know this means there is some sort of "out of
memory" issue, where the Windows architecture is
"too proud" to declare a dumpster fire.

I managed to get Task Manager running.

I see the Reflectbin or whatever running.

I had a chance to kill it, because the machine was
still responding. For some reason, I flipped over to
something else for a look.

And Windows crashed.

The last thing that happened, is a chunk of Firefox screen
was pasted as a texture into the desktop background. So
part of the fun involves the Firefox compositor stepping
outside of its bounds and screwing up the display.

In other words, a "nothing burger" test case,
crashes the machine.

Now I know why I don't run the most modern Firefox :-/

This is hardly an encouraging way to start a set
of test cases. Where do I go from here I wonder ?
Take up stamp collecting ?

One way to moderate this issue, is to mod the cache
settings on the drives. But I'm hardly likely to spend
weeks doing that to every install sitting in the
computer room.

The only other dumpster fire of this sort I've managed
to set in the computer room, was mounting a VHD file
in Win10 and doing some sort of hispeed I/O to it, and
watching as the machine ran out of memory while doing so.
And Windows 10 froze on me (required pusha da button).

You usually only get one tiny opportunity to wiggle
out of these. If you get Task Manager running, you have to
get your priorities right, as you may only get
one chance to stop the thing applying memory pressure.
If you close Task Manager, odds are you can't open it
a second time.

Paul

Frank Slootweg wrote:

VanguardLH wrote:
Frank Slootweg wrote:

FWIW, I use a program (NetWorx [1]) which also uses a sqlite database.
I also use Cobian Backup, do *not* shutdown the program/database and
I do *not* get an error.

[1] https://www.softperfect.com/products/networx

Are you sure Networx does not place any database records, extends, or
delta updates in memory (which is not covered in a file/disk backup)?

https://www.sqlite.org/inmemorydb.html

No, I'm not *sure*, but I do not think so. Networx' sqlite database is
kept in a file, but I cannot rule out that that file is used to fill an
sqlite 'In-Memory Database', which is later written back to the disk
file.

The modification time of the database file is currently 16:50 on July
6, i.e. yesterday. That might imply that the active database is not in
that file, but I don't think Windows updates the modification time of
open files (i.e. only at close or specific request).

N.B. Besides the database file (NetWorx.db), there are two other files
(NetWorx.db-wal and NetWorx.db-shm), where the .db-wal file has nearly
the same timestamp (16:51), but the .db-shm (shared memory?) file is of
June 13 at 11:15, which is the time of the last boot.

Anyway, my example was just one of similar circumstances as T's
scenario (sqlite database being backed up by Cobian Backup), which did
not give an error (from Cobian Backup).

Not sure how you could include a drive in a backup job definition that
is a RAM disk, especially one that is created on-the-fly during
operation, and may contain only some parts of the database that would
benefit most from the speed of RAM instead of being on disk.

https://www.softperfect.com/contact/...php?article=44

AFAICT, sqlite 'In-Memory Databases' are not using a RAM disk, just
normal memory, so I don't see how your point is related to the
scenario(s) at hand.

That said, my compliments for your - as always - very thorough
research!

I don't know that Mozilla's implementation of SQLite involves any memory
components of their databases. I was just mentioning why typical backup
programs may fail when including databases. I know from corporate
backup schemes where MS-SQL is used that the sysadmins didn't just
backup the databases, but had to do some prep beforehand or use backup
programs that specifically supported those databases. I haven't been
involved with helping sysadmins for a long time. I've read that
Microsoft SQL has its own VSS writer, so it can support VSS. SQLite is,
well, a lite variant although it accepts many of the SQL commands.
There's something about SQLite, like it performs only disk-level
transactions, versus SQL which can utilize memory buffers or save deltas
in memory for fast transactions by thousands of concurrent users.

When talking to a tech/dev for eM Client, a local e-mail program (they
also have an app version), and discussing doing backups (file or image)
that included their database, they claimed that upon restore that the
database would be empty or corrupted if eM Client was still running at
the time of the backup. That's why they included their own dbBackup
program to allow backing up the database while leaving eM Client running
all the time.

Although SQLite can put a database into memory for much faster
performance, I haven't heard of Mozilla doing that for Firefox. After
all, adding bookmarks, navigating the web to reflect changes in history,
and other user intervention is pretty slow compared to updating a
disk-based database. But then, so is getting and sending e-mails and
updating calendars and contacts in an e-mail client (e.g., eM Client).

SQLite locks the databases on writes. I don't know how fast it releases
the lock, like if the lock is only during a write or if it expires
after, say, 10 minutes to eliminate the overwrite for many subsequent
writes. Some databases support replication (that is quiescent; i.e., no
writes after replication) to have that backed up. Each VSS-aware
program installs its own VSS writer during its installation. The writer
tells VSS how to backup the program and its data. The VSS writer also
makes sure the data is quiescent (the program doesn't access it) until a
shadow copy is created. After the VSS writer has successfully completed
its pre-backup chores, the VSS requestor flags the VSS provider to
create the snapshot. You'd think having the snapshot was enough, but
apparently after the backup has completed is when the VSS requestor
announces the activity has completed which instructs the VSS writer to
do any post-backup chores, so the program can return to its regular
activities. In a way that makes sense. You certainly wouldn't want to
double the disk footprint and spend time for a copy operation by
creating entire copies of the source data as a snapshot copy that get
copied by a backup program. The snapshot is some info about the data
that has been made quiescent before it can be copied by a backup
program.

https://docs.microsoft.com/en-us/pre...785914(v=ws.10)

Writers: Preventing Data Inconsistencies

Writers are software that is included in applications and services
that help provide consistent shadow copies. Writers serve two main
purposes:

* When applications and services are running, the writer responds to
signals provided by the Volume Shadow Copy Service interface to
allow applications to prepare and quiesce their data stores for
shadow copy creation and to ensure that no writes occur on the
volume while the shadow copy is being created. (During quiescence,
applications make data on the disk consistent. For example, an
application might flush its buffers to disk or write out in-memory
data to disk.)

* The writer also provides information about the application name,
icons, files to include or exclude, and a strategy to restore the
files.

Many backup programs will advertise, for example, that they will include
Outlook PST files in their backups. This is an extra feature they can
brag about. Outlook doesn't have a VSS writer, and why it is
recommended to close Outlook before doing a backup on it. I don't what
trick the Outlook-aware backup programs use to let them ensure they can
save a consistent copy of the PST file (as I recall, OST files are
skipped since the source data is on an IMAP or Exchange server which is
itself the backup). Perhaps one trick is to lock the .pst file to
temporarily keep out Outlook from writing to it, save the file in a
backup, and release the lock. I would think Outlook would generate
errors if it retrieved new e-mails from the server during that time.
Some backup programs cannot get a good or any copy of the PST file, some
can, Outlook doesn't have a VSS writer, so those that can must be using
some trick to forcibly quiesce Outlook's message store. Another
possible trick, and because backup jobs run with admin privileges or
under the SYSTEm account, is to issue a halt on the outlook.exe process
during the backup and release after the backup has completed. If you
use a backup program that is not Outlook capable, restoring the PST file
can result in restoring a corrupted copy. That's why Outlook has its
own integrity verification when it loads the .pst file. If an e-mail in
transit wasn't included in the backup, it get included in the next
backup. Outlook's inability to quiesce its message store is also why
having the .pst file on a network share usually results in later
problems with accessing it and causes corruption of the .pst file
(network connections are not persistent, so when the connection drops
then the .pst file is smashed closed instead of properly finishing the
writes or other command and closing the file).

While "vssadmin list writers" will list the registered VSS writers, only
some are obvious while they rest you have to dig into the registry to
figure out what or whoe they are for. Nothing struck me as obvious for
a VSS writer to use with SQLite.

https://support.carbonite.com/articl...s-and-Services

The "List of VSS Writers" lists some of the VSS writers but is not a
comprehensive list.

While many database programs include their own VSS writer, SQLite does
not, so there is no way to quiesce the SQLite databases. However,
except for if the app uses SQLite in memory, it seems usually the
..sqlite file can be copied. A backup might error on the file (and
proceed to backup other files), but usually when the user initiates a
backup or it is scheduled off-hours, the user isn't web surfing.

VSS is not something that can be forced upon a program to ensure its
data is quiescent to guarantee a consistent or unchanging data state is
available during a backup. VSS is a cooperative affair. If a program
is not VSS aware, and if the program has not interface to allow external
processes to affect its database, tricks are needed to get a safe state
for the data to backup, or the backup program skips those files, or the
backup program saves the files but no assurance they are safe to use
upon a restore.

https://www.sqlite.org/backup.html

SQLite doesn't have a VSS writer, but it does have a command a process,
like a backup program, could issue to lock the database. However, that
means the backup program would have to specifically support SQLite
databases, like it has an ODBC driver for SQLite. That's probably the
same reason eM Client supplies their own dbBackup program since no
backup programs are eM Client aware (and eM Client might not provide the
API for other processes to lock their database).

AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite
Read timed out.
The customer leaves Firefox running 24/7

A backup using Cobian to a local FTP server that normally
takes 20 minutes was taking over 24 hours.

I always close Firefox before backing up my Profile;
I "RoboCopy" backup my entire profile,
and select files are uploaded to my FTP server.

close Firefox before backing up a Profile

customer leaves his computer running 24/7 :'(

Tell the customer that he has to close Firefox.