If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
|
Thread Tools | Rate Thread | Display Modes |
#1
|
|||
|
|||
places.sqlite read time out
Hi All,
Windows Nein, oops Windows Ten - build 1809 Firefox 67.0.4, 64 bit This is from Cobian Backup running Volume Shadow Copy Service (VSS). ERR 2019-07-02 14:07 Error uploading the file "GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite": Read timed out. Is places.sqlite playing tricks with VSS? Are there things VSS can't copy? The customer does leave Firefox running 24/7 Up time was 11 days. Maybe that was the problem. A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. Many thanks, -T |
Ads |
#2
|
|||
|
|||
places.sqlite read time out
Tee, oops, Tea, oops, Ti, oops, T wrote:
Windows Nein, oops Windows Ten - build 1809 Firefox 67.0.4, 64 bit This is from Cobian Backup running Volume Shadow Copy Service (VSS). ERR 2019-07-02 14:07 Error uploading the file "GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite": Read timed out. Is places.sqlite playing tricks with VSS? Are there things VSS can't copy? Yep, I've run across this with eM Client that uses a database. I talked to a dev about leaving eM Client running 24x7 and running backups. He said that VSS would create a shadow copy of the database but the delta changes were saved elsewhere and applied later to the database. He didn't mention another file, and if that were true I would think a backup with both files would be sufficient to restore the message store. I got cut short due to an appointment, but I started to wonder if the delta changes were in memory which means they wouldn't be saved in a backup (whether a file or image backup). He said the result of a shadow copy of the database would result in nothing saved. I had to make sure em Client was closed, so its database was quiescent. Alternatively, the rep said I would have to use their own database backup utility they include in the installation but never identify in their documentation (called dbBackup, as I recall). I've seen similar issues with Microsoft's SQL and other database programs. A shadow copy was not sufficient to thoroughly back it up which meant a restore resulted in a corrupted database. You needed to use a backup program that specifically handled databases. SQLite, mSQL, and SQL have a CLI (command-line interface) that let you run commands, like SELECT. I suspect before backing up the database that you first do a PURGE to physically erase the delete-flagged records from the database, a COMPACT to remove the empty space, and issue a BACKUP command to the SQL server, but that's likely a huge simplification. SQLite lets you issue SQL commands, but I suspect you first have to disconnect Firefox from keeping the database file open or even updating it while open. An SQL database lock is not the same as a file system lock attribute. https://www.sqlite.org/lockingv3.html If the SQLite database is small enough, and to boost its performance, it can be in memory (hence why I wondered a shadow copy of eM Client's database would result in saving a zero or partial database file); see: https://www.sqlite.org/inmemorydb.html That's why I figure the safest way to backup an SQL database is to first make sure it is quiescent (no in use, closed after applying delta updates). Simple databases, like PST for Outlook, can use VSS for backing up, but SQL is a different monster. The customer does leave Firefox running 24/7 Up time was 11 days. Maybe that was the problem. A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. Well, if the web page left loaded in the web browser was dynamic, it keeps changing its content. That's a moving target for a backup. The locally cached copy of the document (aka web page) keeps changing. http://kb.mozillazine.org/Places.sqlite Since many sites use dynamic pages, or even use the meta-refresh to repaint a page, the history keeps updating. Only if the user leave a page loaded loaded (and for all tabs) that is completely static or an internal about:name resource, like about:blank, will the files for the web browser not keep changing. Wonder what is this user's memory footprint for Firefox after leaving it open for many days. |
#3
|
|||
|
|||
Close Firefox before backing up my Profile.
On 7/6/2019 10:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite Read timed out. The customer leaves Firefox running 24/7 A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. I always close Firefox before backing up my Profile; I "RoboCopy" backup my entire profile, and select files are uploaded to my FTP server. Unless you close Firefox, before backing up the profile, there will be files that are missed. To truly back up your entire profile you must close Firefox before starting the back up, so all of the files that Firefox uses are closed when you start the back up. You can verify this fact by going into the prifile, sorting by file date, opening Firefox, doing something and close it. You can see the file times closing to the current time. -- Judge your ancestors by how well they met their standards not yours. They did not know your standards, so could not try to meet them. |
#4
|
|||
|
|||
Close Firefox before backing up my Profile.
Keith Nuttle wrote:
On 7/6/2019 10:02 AM, "Jeff-Relf.Me@."@ wrote: AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite Read timed out. The customer leaves Firefox running 24/7 A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. I always close Firefox before backing up my Profile; I "RoboCopy" backup my entire profile, and select files are uploaded to my FTP server. Unless you close Firefox, before backing up the profile, there will be files that are missed. To truly back up your entire profile you must close Firefox before starting the back up, so all of the files that Firefox uses are closed when you start the back up. You can verify this fact by going into the prifile, sorting by file date, opening Firefox, doing something and close it. You can see the file times closing to the current time. You know that things don't have to work that way. Applications which are "VSS aware" work perfectly fine. As long as nobody at Mozilla gives a rats ass, it is what it is. Using "vssadmin list writers" after a backup has functioned abnormally, will give feedback on what VSS thought of the run. https://kb.datto.com/hc/en-us/articl...ws-Server-2003 There are things that are notoriously difficult to back up. A database running 24/7, there's no practical way to interrupt them. And the command to "quiesce" the database, which has ten seconds to complete, is highly unlikely to work. I don't understand why Firefox couldn't flush the database and quiesce, allowing places.sqlite to be backed up. But that would require Firefox to be VSS aware, and code (that wouldn't be needed on Linux) to be added. Of course, exiting your open applications and backing up, is the lowest risk approach. Even then, the windows.edb of the Search Indexer, is not being backed up by Macrium, which is why the Indexer wastes three hours re-indexing C: when the system comes back up after a restore. Lots of stuff isn't being backed up. And Macrium knows that the materials are not essential (or can't be backed up in any case). Paul |
#5
|
|||
|
|||
Close Firefox before backing up my Profile.
Paul wrote:
Applications which are "VSS aware" work perfectly fine. sqlite is blissfully unaware of VSS, so you'll get 'crash consistent' copies taken while the db file is open, not sure why the backup would think it needed to wait? |
#6
|
|||
|
|||
places.sqlite read time out
T wrote:
Hi All, Windows Nein, oops Windows Ten - build 1809 Firefox 67.0.4, 64 bit This is from Cobian Backup running Volume Shadow Copy Service (VSS). ERR 2019-07-02 14:07 Error uploading the file "GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite": Read timed out. Is places.sqlite playing tricks with VSS? Are there things VSS can't copy? The customer does leave Firefox running 24/7 Up time was 11 days. Maybe that was the problem. A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. Many thanks, FWIW, I use a program (NetWorx [1]) which also uses a sqlite database. I also use Cobian Backup, do *not* shutdown the program/database and I do *not* get an error. [1] https://www.softperfect.com/products/networx |
#7
|
|||
|
|||
Close Firefox before backing up my Profile.
Andy Burns wrote:
Paul wrote: Applications which are "VSS aware" work perfectly fine. sqlite is blissfully unaware of VSS, so you'll get 'crash consistent' copies taken while the db file is open, not sure why the backup would think it needed to wait? VSS manages a certain phase of the backup. 1) Request comes into VSS - "Make a shadow of C: please" 2) VSS sends command to running applications to quiesce. 3) Waits ten seconds for compliance. 4) Anything not answering, doesn't get backed up properly. 5) Results may be reflected in VSS status interface. https://www.sqlite.org/howtocorrupt.html "Backup or restore while a transaction is active Systems that run automatic backups in the background might try to make a backup copy of an SQLite database file while it is in the middle of a transaction. The backup copy then might contain some old and some new content, and thus be corrupt. The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library." https://www.sqlite.org/backup.html My guess would be, Firefox would need to intercept a VSS request, and convert it into a Backup API sequence. Or something. Paul |
#8
|
|||
|
|||
Close Firefox before backing up my Profile.
Keith Nuttle wrote:
On 7/6/2019 10:02 AM, "Jeff-Relf.Me@."@ wrote: AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite Read timed out. The customer leaves Firefox running 24/7 A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. I always close Firefox before backing up my Profile; I "RoboCopy" backup my entire profile, and select files are uploaded to my FTP server. Unless you close Firefox, before backing up the profile, there will be files that are missed. To truly back up your entire profile you must close Firefox before starting the back up, so all of the files that Firefox uses are closed when you start the back up. You can verify this fact by going into the prifile, sorting by file date, opening Firefox, doing something and close it. You can see the file times closing to the current time. Unless you continuously backup each and every changed file, by definition "there will be files that are missed" between subsequent backups. The point in this case is, whether backing up a running instance of Firefox will result in data corruption/loss at restore, which is *worse* than the inherent loss of data due to the interval between backups. For a *webbrowser*, I find that hard to believe. So yes, there will be data loss because of the backup interval, but I strongly doubt that there will be more data loss if Firefox isn't shutdown during the backup. Out of interest: Do you shutdown Thunderbird during backup? And what about any other programs which use some kind of (pseudo) database behind the scenes? FYI, I run a local/private newsserver (Hamster) which (of course) also uses a database. I do not shut it down during backups for the reason given above. BTW, if you use a real, transaction oriented, database and cannot permit to lose any transactions, it's of course a whole different ball game with the need for fail-safe transaction logging, rollback, etc., but that's not the scenario we're discussing here. |
#9
|
|||
|
|||
places.sqlite read time out
On Sat, 6 Jul 2019 09:44:57 -0500, VanguardLH wrote:
Wonder what is this user's memory footprint for Firefox after leaving it open for many days. I don't know about the OP, but I currently have 8 Firefox windows open, with a total of 107 tabs, a very low number for me, most of which have been open since I rebooted (let me check Task Manager) just over 86 days ago. The current memory footprint for Firefox is 1.2GB, but I suspect that would be highly dependent upon exactly which pages are loaded. Not all pages are alike. I plan to reboot at the end of the month, but then again, I sort of planned to reboot at the end of June and it didn't happen. |
#10
|
|||
|
|||
places.sqlite read time out
Frank Slootweg wrote:
FWIW, I use a program (NetWorx [1]) which also uses a sqlite database. I also use Cobian Backup, do *not* shutdown the program/database and I do *not* get an error. [1] https://www.softperfect.com/products/networx Are you sure Networx does not place any database records, extends, or delta updates in memory (which is not covered in a file/disk backup)? https://www.sqlite.org/inmemorydb.html Not sure how you could include a drive in a backup job definition that is a RAM disk, especially one that is created on-the-fly during operation, and may contain only some parts of the database that would benefit most from the speed of RAM instead of being on disk. https://www.softperfect.com/contact/...php?article=44 |
#11
|
|||
|
|||
Close Firefox before backing up my Profile.
On 7/6/19 7:02 AM, "Jeff-Relf.Me@."@ wrote:
AppData\Roaming\Mozilla\Firefox\Profiles\xxx\place s.sqlite Read timed out. The customer leaves Firefox running 24/7 A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. I always close Firefox before backing up my Profile; I "RoboCopy" backup my entire profile, and select files are uploaded to my FTP server. customer leaves his computer running 24/7 :'( |
#12
|
|||
|
|||
Close Firefox before backing up my Profile.
On 7/6/19 9:51 AM, Andy Burns wrote:
Paul wrote: Applications which are "VSS aware" work perfectly fine. sqlite is blissfully unaware of VSS, so you'll get 'crash consistent' copies taken while the db file is open, not sure why the backup would think it needed to wait? Okay. That explains why VSS has an issue. I catch the same problem when folks leave their Quick Books open |
#13
|
|||
|
|||
Close Firefox before backing up my Profile.
On 7/6/19 10:31 AM, Paul wrote:
Andy Burns wrote: Paul wrote: Applications which are "VSS aware" work perfectly fine. sqlite is blissfully unaware of VSS, so you'll get 'crash consistent' copies taken while the db file is open, not sure why the backup would think it needed to wait? VSS manages a certain phase of the backup. 1) Request comes into VSS - "Make a shadow of C: please" 2) VSS sends command to running applications to quiesce. 3) Waits ten seconds for compliance. 4) Anything not answering, doesn't get backed up properly. 5) Results may be reflected in VSS status interface. https://www.sqlite.org/howtocorrupt.html Â* "Backup or restore while a transaction is active Â*Â* Systems that run automatic backups in the background might Â*Â* try to make a backup copy of an SQLite database file while Â*Â* it is in the middle of a transaction. The backup copy then Â*Â* might contain some old and some new content, and thus be corrupt. Â*Â* The best approach to make reliable backup copies of an Â*Â* SQLite database is to make use of the backup API Â*Â* that is part of the SQLite library." https://www.sqlite.org/backup.html My guess would be, Firefox would need to intercept a VSS request, and convert it into a Backup API sequence. Or something. Â*Â* Paul Hi Paul, I am thinking a this point the best approach is a pre-event script: Taskkill /IM firefox.exe wait five seconds Taskkill /IM firefox.exe /F and thuderbird and chrome and quickbooks, etc.. Thank you for the education. You are an excellent teacher. I did not realize that VSS had such limitations. -T |
#14
|
|||
|
|||
places.sqlite read time out
On 7/6/19 7:44 AM, VanguardLH wrote:
Tee, oops, Tea, oops, Ti, oops, T wrote: Windows Nein, oops Windows Ten - build 1809 Firefox 67.0.4, 64 bit This is from Cobian Backup running Volume Shadow Copy Service (VSS). ERR 2019-07-02 14:07 Error uploading the file "GLOBALROOT\Device\HarddiskVolumeShadowCopy12\User s\Matt\AppData\Roaming\Mozilla\Firefox\Profiles\zi wxc4ke.default\places.sqlite": Read timed out. Is places.sqlite playing tricks with VSS? Are there things VSS can't copy? Yep, I've run across this with eM Client that uses a database. I talked to a dev about leaving eM Client running 24x7 and running backups. He said that VSS would create a shadow copy of the database but the delta changes were saved elsewhere and applied later to the database. He didn't mention another file, and if that were true I would think a backup with both files would be sufficient to restore the message store. I got cut short due to an appointment, but I started to wonder if the delta changes were in memory which means they wouldn't be saved in a backup (whether a file or image backup). He said the result of a shadow copy of the database would result in nothing saved. I had to make sure em Client was closed, so its database was quiescent. Alternatively, the rep said I would have to use their own database backup utility they include in the installation but never identify in their documentation (called dbBackup, as I recall). I've seen similar issues with Microsoft's SQL and other database programs. A shadow copy was not sufficient to thoroughly back it up which meant a restore resulted in a corrupted database. You needed to use a backup program that specifically handled databases. SQLite, mSQL, and SQL have a CLI (command-line interface) that let you run commands, like SELECT. I suspect before backing up the database that you first do a PURGE to physically erase the delete-flagged records from the database, a COMPACT to remove the empty space, and issue a BACKUP command to the SQL server, but that's likely a huge simplification. SQLite lets you issue SQL commands, but I suspect you first have to disconnect Firefox from keeping the database file open or even updating it while open. An SQL database lock is not the same as a file system lock attribute. https://www.sqlite.org/lockingv3.html If the SQLite database is small enough, and to boost its performance, it can be in memory (hence why I wondered a shadow copy of eM Client's database would result in saving a zero or partial database file); see: https://www.sqlite.org/inmemorydb.html That's why I figure the safest way to backup an SQL database is to first make sure it is quiescent (no in use, closed after applying delta updates). Simple databases, like PST for Outlook, can use VSS for backing up, but SQL is a different monster. The customer does leave Firefox running 24/7 Up time was 11 days. Maybe that was the problem. A backup using Cobian to a local FTP server that normally takes 20 minutes was taking over 24 hours. Well, if the web page left loaded in the web browser was dynamic, it keeps changing its content. That's a moving target for a backup. The locally cached copy of the document (aka web page) keeps changing. http://kb.mozillazine.org/Places.sqlite Since many sites use dynamic pages, or even use the meta-refresh to repaint a page, the history keeps updating. Only if the user leave a page loaded loaded (and for all tabs) that is completely static or an internal about:name resource, like about:blank, will the files for the web browser not keep changing. Wonder what is this user's memory footprint for Firefox after leaving it open for many days. I am mainly concerned about the saved password and account. Thank you for the excellent write up! -T |
#15
|
|||
|
|||
Close Firefox before backing up my Profile.
T wrote:
On 7/6/19 10:31 AM, Paul wrote: Andy Burns wrote: Paul wrote: Applications which are "VSS aware" work perfectly fine. sqlite is blissfully unaware of VSS, so you'll get 'crash consistent' copies taken while the db file is open, not sure why the backup would think it needed to wait? VSS manages a certain phase of the backup. 1) Request comes into VSS - "Make a shadow of C: please" 2) VSS sends command to running applications to quiesce. 3) Waits ten seconds for compliance. 4) Anything not answering, doesn't get backed up properly. 5) Results may be reflected in VSS status interface. https://www.sqlite.org/howtocorrupt.html "Backup or restore while a transaction is active Systems that run automatic backups in the background might try to make a backup copy of an SQLite database file while it is in the middle of a transaction. The backup copy then might contain some old and some new content, and thus be corrupt. The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library." https://www.sqlite.org/backup.html My guess would be, Firefox would need to intercept a VSS request, and convert it into a Backup API sequence. Or something. Paul Hi Paul, I am thinking a this point the best approach is a pre-event script: Taskkill /IM firefox.exe wait five seconds Taskkill /IM firefox.exe /F and thuderbird and chrome and quickbooks, etc.. Thank you for the education. You are an excellent teacher. I did not realize that VSS had such limitations. -T VSS is, in a sense, a "cooperative" system. It delivers a hint to your program. If you listen and behave in a certain way, your "goods" get properly backed up. No home programmer puts a VSS handler in their code, because if a home programmer makes a program, they probably "exit" before doing their backup. With "scheduled" backups, I suppose it could be different, because the human may not be present for "cleanup". Databases (especially the standalone big ones) are inherently difficult customers to deal with. It's never going to go well with them. But the way that Firefox uses sqlite, has possibilities. Firefox is not a five-nines server. It can afford to shut off the databases, flush them, for short intervals. And it should be possible to make it play nice. Once the shadow copy is made, of a snapshot-in-time, the backup program then can back up the shadow "snapshot" at its leisure. That's the beauty of the ten second interval. Be quiet for ten seconds, close the file, "Click" goes the snapshot, now open your file again and continue on. Once VSS makes the snapshot, that's the limit of the intrusion. Thunderbird is more problematic, in that seemingly random stuff happens in there. There is less "order" in a Thunderbird universe, at least to my "observer eye". The more modern the version of Thunderbird, the more wacky the behavior. I could well imagine Thunderbird lighting up some sqlites in the middle of a VSS call. Just because it has no internal synchronization primitives. If you ever watch the random order of the newsgroups it works on, you'll get the idea. Paul |
|
Thread Tools | |
Display Modes | Rate This Thread |
|
|