A Windows XP help forum. PCbanter

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » PCbanter forum » Windows 10 » Windows 10 Help Forum
Site Map Home Register Authors List Search Today's Posts Mark Forums Read Web Partners

places.sqlite read time out



 
 
Thread Tools Rate Thread Display Modes
  #1  
Old July 6th 19, 09:32 AM posted to alt.comp.os.windows-10
T
external usenet poster
 
Posts: 4,600
Default places.sqlite read time out

Hi All,

Windows Nein, oops Windows Ten - build 1809
Firefox 67.0.4, 64 bit

This is from Cobian Backup running Volume Shadow Copy Service (VSS).

ERR 2019-07-02 14:07 Error uploading the file
"GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite":
Read timed out.

Is places.sqlite playing tricks with VSS? Are there things
VSS can't copy?

The customer does leave Firefox running 24/7

Up time was 11 days. Maybe that was the problem. A backup
using Cobian to a local FTP server that normally takes 20
minutes was taking over 24 hours.

Many thanks,
-T
Ads
  #2  
Old July 6th 19, 03:44 PM posted to alt.comp.os.windows-10
VanguardLH[_2_]
external usenet poster
 
Posts: 10,881
Default places.sqlite read time out

Tee, oops, Tea, oops, Ti, oops, T wrote:

Windows Nein, oops Windows Ten - build 1809
Firefox 67.0.4, 64 bit

This is from Cobian Backup running Volume Shadow Copy Service (VSS).

ERR 2019-07-02 14:07 Error uploading the file
"GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite":
Read timed out.

Is places.sqlite playing tricks with VSS? Are there things
VSS can't copy?


Yep, I've run across this with eM Client that uses a database. I talked
to a dev about leaving eM Client running 24x7 and running backups. He
said that VSS would create a shadow copy of the database but the delta
changes were saved elsewhere and applied later to the database. He
didn't mention another file, and if that were true I would think a
backup with both files would be sufficient to restore the message store.
I got cut short due to an appointment, but I started to wonder if the
delta changes were in memory which means they wouldn't be saved in a
backup (whether a file or image backup). He said the result of a shadow
copy of the database would result in nothing saved. I had to make sure
em Client was closed, so its database was quiescent. Alternatively, the
rep said I would have to use their own database backup utility they
include in the installation but never identify in their documentation
(called dbBackup, as I recall).

I've seen similar issues with Microsoft's SQL and other database
programs. A shadow copy was not sufficient to thoroughly back it up
which meant a restore resulted in a corrupted database. You needed to
use a backup program that specifically handled databases.

SQLite, mSQL, and SQL have a CLI (command-line interface) that let you
run commands, like SELECT. I suspect before backing up the database
that you first do a PURGE to physically erase the delete-flagged records
from the database, a COMPACT to remove the empty space, and issue a
BACKUP command to the SQL server, but that's likely a huge
simplification. SQLite lets you issue SQL commands, but I suspect you
first have to disconnect Firefox from keeping the database file open or
even updating it while open.

An SQL database lock is not the same as a file system lock attribute.

https://www.sqlite.org/lockingv3.html

If the SQLite database is small enough, and to boost its performance, it
can be in memory (hence why I wondered a shadow copy of eM Client's
database would result in saving a zero or partial database file); see:

https://www.sqlite.org/inmemorydb.html

That's why I figure the safest way to backup an SQL database is to first
make sure it is quiescent (no in use, closed after applying delta
updates). Simple databases, like PST for Outlook, can use VSS for
backing up, but SQL is a different monster.

The customer does leave Firefox running 24/7

Up time was 11 days. Maybe that was the problem. A backup
using Cobian to a local FTP server that normally takes 20
minutes was taking over 24 hours.


Well, if the web page left loaded in the web browser was dynamic, it
keeps changing its content. That's a moving target for a backup. The
locally cached copy of the document (aka web page) keeps changing.

http://kb.mozillazine.org/Places.sqlite

Since many sites use dynamic pages, or even use the meta-refresh to
repaint a page, the history keeps updating. Only if the user leave a
page loaded loaded (and for all tabs) that is completely static or an
internal about:name resource, like about:blank, will the files for the
web browser not keep changing.

Wonder what is this user's memory footprint for Firefox after leaving it
open for many days.
  #3  
Old July 6th 19, 04:13 PM posted to alt.comp.os.windows-10
Keith Nuttle
external usenet poster
 
Posts: 1,844
Default Close Firefox before backing up my Profile.

On 7/6/2019 10:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite
Read timed out.
The customer leaves Firefox running 24/7

A backup using Cobian to a local FTP server that normally
takes 20 minutes was taking over 24 hours.


I always close Firefox before backing up my Profile;
I "RoboCopy" backup my entire profile,
and select files are uploaded to my FTP server.

Unless you close Firefox, before backing up the profile, there will be
files that are missed. To truly back up your entire profile you must
close Firefox before starting the back up, so all of the files that
Firefox uses are closed when you start the back up.

You can verify this fact by going into the prifile, sorting by file
date, opening Firefox, doing something and close it. You can see the
file times closing to the current time.

--
Judge your ancestors by how well they met their standards not yours.
They did not know your standards, so could not try to meet them.
  #4  
Old July 6th 19, 05:24 PM posted to alt.comp.os.windows-10
Paul[_32_]
external usenet poster
 
Posts: 11,873
Default Close Firefox before backing up my Profile.

Keith Nuttle wrote:
On 7/6/2019 10:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite
Read timed out.
The customer leaves Firefox running 24/7

A backup using Cobian to a local FTP server that normally
takes 20 minutes was taking over 24 hours.


I always close Firefox before backing up my Profile;
I "RoboCopy" backup my entire profile,
and select files are uploaded to my FTP server.

Unless you close Firefox, before backing up the profile, there will be
files that are missed. To truly back up your entire profile you must
close Firefox before starting the back up, so all of the files that
Firefox uses are closed when you start the back up.

You can verify this fact by going into the prifile, sorting by file
date, opening Firefox, doing something and close it. You can see the
file times closing to the current time.


You know that things don't have to work that way.

Applications which are "VSS aware" work perfectly fine.

As long as nobody at Mozilla gives a rats ass,
it is what it is.

Using "vssadmin list writers" after a backup has functioned
abnormally, will give feedback on what VSS thought of the run.

https://kb.datto.com/hc/en-us/articl...ws-Server-2003

There are things that are notoriously difficult to back up.
A database running 24/7, there's no practical way to interrupt
them. And the command to "quiesce" the database, which
has ten seconds to complete, is highly unlikely to work.

I don't understand why Firefox couldn't flush the
database and quiesce, allowing places.sqlite to be
backed up. But that would require Firefox to be VSS aware,
and code (that wouldn't be needed on Linux) to be added.

Of course, exiting your open applications and backing up,
is the lowest risk approach. Even then, the windows.edb
of the Search Indexer, is not being backed up by Macrium,
which is why the Indexer wastes three hours re-indexing
C: when the system comes back up after a restore. Lots
of stuff isn't being backed up. And Macrium knows that
the materials are not essential (or can't be backed up
in any case).

Paul
  #5  
Old July 6th 19, 05:51 PM posted to alt.comp.os.windows-10
Andy Burns[_6_]
external usenet poster
 
Posts: 1,318
Default Close Firefox before backing up my Profile.

Paul wrote:

Applications which are "VSS aware" work perfectly fine.


sqlite is blissfully unaware of VSS, so you'll get 'crash consistent'
copies taken while the db file is open, not sure why the backup would
think it needed to wait?
  #6  
Old July 6th 19, 06:13 PM posted to alt.comp.os.windows-10
Frank Slootweg
external usenet poster
 
Posts: 1,226
Default places.sqlite read time out

T wrote:
Hi All,

Windows Nein, oops Windows Ten - build 1809
Firefox 67.0.4, 64 bit

This is from Cobian Backup running Volume Shadow Copy Service (VSS).

ERR 2019-07-02 14:07 Error uploading the file
"GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite":
Read timed out.

Is places.sqlite playing tricks with VSS? Are there things
VSS can't copy?

The customer does leave Firefox running 24/7

Up time was 11 days. Maybe that was the problem. A backup
using Cobian to a local FTP server that normally takes 20
minutes was taking over 24 hours.

Many thanks,


FWIW, I use a program (NetWorx [1]) which also uses a sqlite database. I
also use Cobian Backup, do *not* shutdown the program/database and I do
*not* get an error.

[1] https://www.softperfect.com/products/networx
  #7  
Old July 6th 19, 06:31 PM posted to alt.comp.os.windows-10
Paul[_32_]
external usenet poster
 
Posts: 11,873
Default Close Firefox before backing up my Profile.

Andy Burns wrote:
Paul wrote:

Applications which are "VSS aware" work perfectly fine.


sqlite is blissfully unaware of VSS, so you'll get 'crash consistent'
copies taken while the db file is open, not sure why the backup would
think it needed to wait?


VSS manages a certain phase of the backup.

1) Request comes into VSS - "Make a shadow of C: please"
2) VSS sends command to running applications to quiesce.
3) Waits ten seconds for compliance.
4) Anything not answering, doesn't get backed up properly.
5) Results may be reflected in VSS status interface.

https://www.sqlite.org/howtocorrupt.html

"Backup or restore while a transaction is active

Systems that run automatic backups in the background might
try to make a backup copy of an SQLite database file while
it is in the middle of a transaction. The backup copy then
might contain some old and some new content, and thus be corrupt.

The best approach to make reliable backup copies of an
SQLite database is to make use of the backup API
that is part of the SQLite library."

https://www.sqlite.org/backup.html

My guess would be, Firefox would need to intercept a VSS
request, and convert it into a Backup API sequence.
Or something.

Paul
  #8  
Old July 6th 19, 06:36 PM posted to alt.comp.os.windows-10
Frank Slootweg
external usenet poster
 
Posts: 1,226
Default Close Firefox before backing up my Profile.

Keith Nuttle wrote:
On 7/6/2019 10:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite
Read timed out.
The customer leaves Firefox running 24/7

A backup using Cobian to a local FTP server that normally
takes 20 minutes was taking over 24 hours.


I always close Firefox before backing up my Profile;
I "RoboCopy" backup my entire profile,
and select files are uploaded to my FTP server.

Unless you close Firefox, before backing up the profile, there will be
files that are missed. To truly back up your entire profile you must
close Firefox before starting the back up, so all of the files that
Firefox uses are closed when you start the back up.

You can verify this fact by going into the prifile, sorting by file
date, opening Firefox, doing something and close it. You can see the
file times closing to the current time.


Unless you continuously backup each and every changed file, by
definition "there will be files that are missed" between subsequent
backups.

The point in this case is, whether backing up a running instance of
Firefox will result in data corruption/loss at restore, which is *worse*
than the inherent loss of data due to the interval between backups. For
a *webbrowser*, I find that hard to believe.

So yes, there will be data loss because of the backup interval, but I
strongly doubt that there will be more data loss if Firefox isn't
shutdown during the backup.

Out of interest: Do you shutdown Thunderbird during backup? And what
about any other programs which use some kind of (pseudo) database behind
the scenes?

FYI, I run a local/private newsserver (Hamster) which (of course) also
uses a database. I do not shut it down during backups for the reason
given above.

BTW, if you use a real, transaction oriented, database and cannot
permit to lose any transactions, it's of course a whole different ball
game with the need for fail-safe transaction logging, rollback, etc.,
but that's not the scenario we're discussing here.
  #9  
Old July 6th 19, 09:34 PM posted to alt.comp.os.windows-10
Char Jackson
external usenet poster
 
Posts: 10,449
Default places.sqlite read time out

On Sat, 6 Jul 2019 09:44:57 -0500, VanguardLH wrote:

Wonder what is this user's memory footprint for Firefox after leaving it
open for many days.


I don't know about the OP, but I currently have 8 Firefox windows open,
with a total of 107 tabs, a very low number for me, most of which have
been open since I rebooted (let me check Task Manager) just over 86 days
ago. The current memory footprint for Firefox is 1.2GB, but I suspect
that would be highly dependent upon exactly which pages are loaded. Not
all pages are alike.

I plan to reboot at the end of the month, but then again, I sort of
planned to reboot at the end of June and it didn't happen.

  #10  
Old July 7th 19, 01:14 AM posted to alt.comp.os.windows-10
VanguardLH[_2_]
external usenet poster
 
Posts: 10,881
Default places.sqlite read time out

Frank Slootweg wrote:

FWIW, I use a program (NetWorx [1]) which also uses a sqlite database.
I also use Cobian Backup, do *not* shutdown the program/database and
I do *not* get an error.

[1] https://www.softperfect.com/products/networx


Are you sure Networx does not place any database records, extends, or
delta updates in memory (which is not covered in a file/disk backup)?

https://www.sqlite.org/inmemorydb.html

Not sure how you could include a drive in a backup job definition that
is a RAM disk, especially one that is created on-the-fly during
operation, and may contain only some parts of the database that would
benefit most from the speed of RAM instead of being on disk.

https://www.softperfect.com/contact/...php?article=44
  #11  
Old July 7th 19, 02:12 AM posted to alt.comp.os.windows-10
T
external usenet poster
 
Posts: 4,600
Default Close Firefox before backing up my Profile.

On 7/6/19 7:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite
Read timed out.
The customer leaves Firefox running 24/7

A backup using Cobian to a local FTP server that normally
takes 20 minutes was taking over 24 hours.


I always close Firefox before backing up my Profile;
I "RoboCopy" backup my entire profile,
and select files are uploaded to my FTP server.


customer leaves his computer running 24/7

:'(


  #12  
Old July 7th 19, 02:14 AM posted to alt.comp.os.windows-10
T
external usenet poster
 
Posts: 4,600
Default Close Firefox before backing up my Profile.

On 7/6/19 9:51 AM, Andy Burns wrote:
Paul wrote:

Applications which are "VSS aware" work perfectly fine.


sqlite is blissfully unaware of VSS, so you'll get 'crash consistent'
copies taken while the db file is open, not sure why the backup would
think it needed to wait?


Okay. That explains why VSS has an issue.

I catch the same problem when folks leave their Quick Books open
  #13  
Old July 7th 19, 02:18 AM posted to alt.comp.os.windows-10
T
external usenet poster
 
Posts: 4,600
Default Close Firefox before backing up my Profile.

On 7/6/19 10:31 AM, Paul wrote:
Andy Burns wrote:
Paul wrote:

Applications which are "VSS aware" work perfectly fine.


sqlite is blissfully unaware of VSS, so you'll get 'crash consistent'
copies taken while the db file is open, not sure why the backup would
think it needed to wait?


VSS manages a certain phase of the backup.

1) Request comes into VSS - "Make a shadow of C: please"
2) VSS sends command to running applications to quiesce.
3) Waits ten seconds for compliance.
4) Anything not answering, doesn't get backed up properly.
5) Results may be reflected in VSS status interface.

https://www.sqlite.org/howtocorrupt.html

Â* "Backup or restore while a transaction is active

Â*Â* Systems that run automatic backups in the background might
Â*Â* try to make a backup copy of an SQLite database file while
Â*Â* it is in the middle of a transaction. The backup copy then
Â*Â* might contain some old and some new content, and thus be corrupt.

Â*Â* The best approach to make reliable backup copies of an
Â*Â* SQLite database is to make use of the backup API
Â*Â* that is part of the SQLite library."

https://www.sqlite.org/backup.html

My guess would be, Firefox would need to intercept a VSS
request, and convert it into a Backup API sequence.
Or something.

Â*Â* Paul


Hi Paul,

I am thinking a this point the best approach is a
pre-event script:

Taskkill /IM firefox.exe
wait five seconds
Taskkill /IM firefox.exe /F

and thuderbird and chrome and quickbooks, etc..

Thank you for the education. You are an excellent teacher.
I did not realize that VSS had such limitations.

-T

  #14  
Old July 7th 19, 02:20 AM posted to alt.comp.os.windows-10
T
external usenet poster
 
Posts: 4,600
Default places.sqlite read time out

On 7/6/19 7:44 AM, VanguardLH wrote:
Tee, oops, Tea, oops, Ti, oops, T wrote:

Windows Nein, oops Windows Ten - build 1809
Firefox 67.0.4, 64 bit

This is from Cobian Backup running Volume Shadow Copy Service (VSS).

ERR 2019-07-02 14:07 Error uploading the file
"GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite":
Read timed out.

Is places.sqlite playing tricks with VSS? Are there things
VSS can't copy?


Yep, I've run across this with eM Client that uses a database. I talked
to a dev about leaving eM Client running 24x7 and running backups. He
said that VSS would create a shadow copy of the database but the delta
changes were saved elsewhere and applied later to the database. He
didn't mention another file, and if that were true I would think a
backup with both files would be sufficient to restore the message store.
I got cut short due to an appointment, but I started to wonder if the
delta changes were in memory which means they wouldn't be saved in a
backup (whether a file or image backup). He said the result of a shadow
copy of the database would result in nothing saved. I had to make sure
em Client was closed, so its database was quiescent. Alternatively, the
rep said I would have to use their own database backup utility they
include in the installation but never identify in their documentation
(called dbBackup, as I recall).

I've seen similar issues with Microsoft's SQL and other database
programs. A shadow copy was not sufficient to thoroughly back it up
which meant a restore resulted in a corrupted database. You needed to
use a backup program that specifically handled databases.

SQLite, mSQL, and SQL have a CLI (command-line interface) that let you
run commands, like SELECT. I suspect before backing up the database
that you first do a PURGE to physically erase the delete-flagged records
from the database, a COMPACT to remove the empty space, and issue a
BACKUP command to the SQL server, but that's likely a huge
simplification. SQLite lets you issue SQL commands, but I suspect you
first have to disconnect Firefox from keeping the database file open or
even updating it while open.

An SQL database lock is not the same as a file system lock attribute.

https://www.sqlite.org/lockingv3.html

If the SQLite database is small enough, and to boost its performance, it
can be in memory (hence why I wondered a shadow copy of eM Client's
database would result in saving a zero or partial database file); see:

https://www.sqlite.org/inmemorydb.html

That's why I figure the safest way to backup an SQL database is to first
make sure it is quiescent (no in use, closed after applying delta
updates). Simple databases, like PST for Outlook, can use VSS for
backing up, but SQL is a different monster.

The customer does leave Firefox running 24/7

Up time was 11 days. Maybe that was the problem. A backup
using Cobian to a local FTP server that normally takes 20
minutes was taking over 24 hours.


Well, if the web page left loaded in the web browser was dynamic, it
keeps changing its content. That's a moving target for a backup. The
locally cached copy of the document (aka web page) keeps changing.

http://kb.mozillazine.org/Places.sqlite

Since many sites use dynamic pages, or even use the meta-refresh to
repaint a page, the history keeps updating. Only if the user leave a
page loaded loaded (and for all tabs) that is completely static or an
internal about:name resource, like about:blank, will the files for the
web browser not keep changing.

Wonder what is this user's memory footprint for Firefox after leaving it
open for many days.


I am mainly concerned about the saved password and account.

Thank you for the excellent write up!

-T
  #15  
Old July 7th 19, 02:53 AM posted to alt.comp.os.windows-10
Paul[_32_]
external usenet poster
 
Posts: 11,873
Default Close Firefox before backing up my Profile.

T wrote:
On 7/6/19 10:31 AM, Paul wrote:
Andy Burns wrote:
Paul wrote:

Applications which are "VSS aware" work perfectly fine.

sqlite is blissfully unaware of VSS, so you'll get 'crash consistent'
copies taken while the db file is open, not sure why the backup would
think it needed to wait?


VSS manages a certain phase of the backup.

1) Request comes into VSS - "Make a shadow of C: please"
2) VSS sends command to running applications to quiesce.
3) Waits ten seconds for compliance.
4) Anything not answering, doesn't get backed up properly.
5) Results may be reflected in VSS status interface.

https://www.sqlite.org/howtocorrupt.html

"Backup or restore while a transaction is active

Systems that run automatic backups in the background might
try to make a backup copy of an SQLite database file while
it is in the middle of a transaction. The backup copy then
might contain some old and some new content, and thus be corrupt.

The best approach to make reliable backup copies of an
SQLite database is to make use of the backup API
that is part of the SQLite library."

https://www.sqlite.org/backup.html

My guess would be, Firefox would need to intercept a VSS
request, and convert it into a Backup API sequence.
Or something.

Paul


Hi Paul,

I am thinking a this point the best approach is a
pre-event script:

Taskkill /IM firefox.exe
wait five seconds
Taskkill /IM firefox.exe /F

and thuderbird and chrome and quickbooks, etc..

Thank you for the education. You are an excellent teacher.
I did not realize that VSS had such limitations.

-T


VSS is, in a sense, a "cooperative" system. It delivers
a hint to your program. If you listen and behave in a
certain way, your "goods" get properly backed up.

No home programmer puts a VSS handler in their
code, because if a home programmer makes a program,
they probably "exit" before doing their backup.

With "scheduled" backups, I suppose it could be
different, because the human may not be present
for "cleanup".

Databases (especially the standalone big ones) are
inherently difficult customers to deal with. It's
never going to go well with them. But the way that
Firefox uses sqlite, has possibilities. Firefox is
not a five-nines server. It can afford to shut off
the databases, flush them, for short intervals. And
it should be possible to make it play nice. Once the
shadow copy is made, of a snapshot-in-time, the
backup program then can back up the shadow "snapshot"
at its leisure. That's the beauty of the ten second
interval. Be quiet for ten seconds, close the file,
"Click" goes the snapshot, now open your file again
and continue on. Once VSS makes the snapshot, that's
the limit of the intrusion.

Thunderbird is more problematic, in that seemingly
random stuff happens in there. There is less "order"
in a Thunderbird universe, at least to my
"observer eye". The more modern the version of
Thunderbird, the more wacky the behavior. I could
well imagine Thunderbird lighting up some sqlites
in the middle of a VSS call. Just because it has
no internal synchronization primitives. If you
ever watch the random order of the newsgroups it
works on, you'll get the idea.

Paul
 




Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off






All times are GMT +1. The time now is 11:20 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 PCbanter.
The comments are property of their posters.