Force calibre portable to sync database without closing the program?

AntonGolubev · 07-31-2018, 09:19 AM

Dear calibre users,

I keep my library in the cloud store (OneDrive) and use calibre-portable to access my library from any PC. However, calibre syncs its database (flushes changes) to disk only, when it is get closed (at least that what I observe). So, I routinely loose my changes in the database, if I keep the program open on one PC (e.g. over night) and add something or edit on another PC, and then close calibre on the first one.

Sequence:
1. Open calibre on PC1. Add a book. Keep the program running.
2. Open same calibre library on another PC and add some books. 2 PC does not see the changes of the PC1. Close calibre.
3. Get back to PC1 and re-open calibre: changes from PC1 are gone.

It is not the sync issue of the cloud file storage (e.g. OneDrive). Calibre either do not flush writes or not closes the file after modifying the database.

Does anybody know how to work around this? I still want to be able to keep a single calibre library, which I would like to manage from all my devices.

Is is possible to force calibre to save its catalog (metadata), without closing the program?

I am using the latest version of calibre portable on latest Windows 10.

Kind regards,
Anton Golubev

kovidgoyal · 07-31-2018, 10:21 AM

calibre most definitely writes changes to the database after every operation. And do not use dropbox and calibre at the same time, you **will** lose data. https://manual.calibre-ebook.com/faq...rked-drive-nas

AntonGolubev · 08-01-2018, 12:53 PM

Hi Kovid,

Thank you for replying and the link! I see the point, but I possible the things can be improved with little effort.

Is it possible to force running Calibre to re-load the database from the filesystem, forgetting his state? E.g. if I forgot to close it on one PC, to make it not overwrite the changes, which another instance already introduces to the disk?

Kind regards,
Anton Golubev

itimpi · 08-01-2018, 01:20 PM

Quote:

Originally Posted by AntonGolubev

Hi Kovid,

Thank you for replying and the link! I see the point, but I possible the things can be improved with little effort.

Is it possible to force running Calibre to re-load the database from the filesystem, forgetting his state? E.g. if I forgot to close it on one PC, to make it not overwrite the changes, which another instance already introduces to the disk?

Kind regards,
Anton Golubev

i think you will find that the problem is on the syncing software in that it does not do the sync until Calibre is shutdown and closes the database! This means that if you start another Calibre instance on a different machine it does not know about the changes made on the still running instance so the two instances end up trampling on each other’s changes.

kovidgoyal · 08-02-2018, 12:05 AM

No, you simply cannot have two unrelated programs writing to the databse file at the same time. THere is no way that can be made robust.

DNSB · 08-02-2018, 02:02 AM

Quote:

Originally Posted by AntonGolubev

Hi Kovid,

Thank you for replying and the link! I see the point, but I possible the things can be improved with little effort.

Is it possible to force running Calibre to re-load the database from the filesystem, forgetting his state? E.g. if I forgot to close it on one PC, to make it not overwrite the changes, which another instance already introduces to the disk?

Kind regards,
Anton Golubev

I don't think anything that Kovid could do would help you when your file sync software will not update the database on the cloud when the file is still open. To quote from the OneDrive for Business restrictions and limitations:

Quote:

Open files can't be synced
A file that’s currently opened by an application can't be synced by OneDrive for Business. To sync the file, close any application where the file is currently being used, and then sync the file.

I tested this behaviour with OneDrive (not for Business) and the same behaviour -- open files were not synced. -- was observed.

I suppose Kovid could reverse engineer the way that Microsoft Office 365/Office 1016 allows some multi-user access to files on OneDrive for Business but I think that would be an incredible waste of his time and talent.

Since you are using calibre portable, I would suggest instead that you use a large enough USB drive to store your calibre library and related files. Sneakernet so to speak.

AntonGolubev · 08-17-2018, 05:29 AM

Hi David,

Thank you for confirming the behavior and your research. Here is exactly my point: let Calibre close his database. Then OneDrive will pick-up changes from the cloud. Calibre then can reload the database. You can see it like a "forget & refresh" button in the GUI. These are all existing functions of Calibre - open, write state, close DB, so there is no much new code to write. And nobody speaking on reversing the OneDrive code. Cloud syncs in this aspect behave all the same - file open for write -> no sync.

Cloud people like me will be happy.

That's the whole point of cloud vs carrying the USB stick - you always have all your data with you (and safe).

Kind regards,
Anton

07-31-2018, 09:19 AM	#1
AntonGolubev Junior Member Posts: 6 Karma: 10 Join Date: Jul 2018 Device: Kindle, PocketBook	Force calibre portable to sync database without closing the program? Dear calibre users, I keep my library in the cloud store (OneDrive) and use calibre-portable to access my library from any PC. However, calibre syncs its database (flushes changes) to disk only, when it is get closed (at least that what I observe). So, I routinely loose my changes in the database, if I keep the program open on one PC (e.g. over night) and add something or edit on another PC, and then close calibre on the first one. Sequence: 1. Open calibre on PC1. Add a book. Keep the program running. 2. Open same calibre library on another PC and add some books. 2 PC does not see the changes of the PC1. Close calibre. 3. Get back to PC1 and re-open calibre: changes from PC1 are gone. It is not the sync issue of the cloud file storage (e.g. OneDrive). Calibre either do not flush writes or not closes the file after modifying the database. Does anybody know how to work around this? I still want to be able to keep a single calibre library, which I would like to manage from all my devices. Is is possible to force calibre to save its catalog (metadata), without closing the program? I am using the latest version of calibre portable on latest Windows 10. Kind regards, Anton Golubev

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre database corrupted or missing with Dropbox sync	shootist	Calibre Companion	9	09-14-2015 02:03 PM
Problem including calibre portable in SyMenu program suite	epik	Devices	7	07-07-2015 04:45 PM
HELP Portable Calibre will not read standard Calible database	ayoshak	Library Management	17	02-04-2014 07:58 PM
sync calibre and itunes-database	Henning	Devices	4	03-23-2012 08:00 AM
iPad sync Stanza library to Calibre database?	vonCZ	Apple Devices	2	05-04-2011 04:59 PM

07-31-2018, 10:21 AM	#2
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	calibre most definitely writes changes to the database after every operation. And do not use dropbox and calibre at the same time, you will lose data. https://manual.calibre-ebook.com/faq...rked-drive-nas

08-01-2018, 12:53 PM	#3
AntonGolubev Junior Member Posts: 6 Karma: 10 Join Date: Jul 2018 Device: Kindle, PocketBook	Hi Kovid, Thank you for replying and the link! I see the point, but I possible the things can be improved with little effort. Is it possible to force running Calibre to re-load the database from the filesystem, forgetting his state? E.g. if I forgot to close it on one PC, to make it not overwrite the changes, which another instance already introduces to the disk? Kind regards, Anton Golubev

08-02-2018, 12:05 AM	#5
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, you simply cannot have two unrelated programs writing to the databse file at the same time. THere is no way that can be made robust.

08-17-2018, 05:29 AM	#7
AntonGolubev Junior Member Posts: 6 Karma: 10 Join Date: Jul 2018 Device: Kindle, PocketBook	Hi David, Thank you for confirming the behavior and your research. Here is exactly my point: let Calibre close his database. Then OneDrive will pick-up changes from the cloud. Calibre then can reload the database. You can see it like a "forget & refresh" button in the GUI. These are all existing functions of Calibre - open, write state, close DB, so there is no much new code to write. And nobody speaking on reversing the OneDrive code. Cloud syncs in this aspect behave all the same - file open for write -> no sync. Cloud people like me will be happy. That's the whole point of cloud vs carrying the USB stick - you always have all your data with you (and safe). Kind regards, Anton