Launching E-book viewer restores deleted annotations / highlights

LuisNeto · 01-14-2024, 10:55 AM

I am working on the development of a program for importing highlights from Kobo e-readers into calibre, and I've encountered a problem due to how calibre manages deleted annotations.

This program works by inserting the highlights into calibre's Annotations database table.
When I open calibre, I can see the highlights in "Browse annotations" and the E-book viewer.

However, I start getting problems after deleting highlights in calibre.

When a highlight is deleted, calibre updates the value of annot_data with an indication that it has been removed.

When I run the program to import the highlights again, it gets this database record, verifies that its JSON in annot_data (which says it's removed) is different from the JSON containing the highlight data from the e-reader, so it updates the database record with the highlight data again, overwriting the JSON that calibre had written (which indicated it was removed).

When I launch calibre, if I click in "Browse annotations" I can see the highlight.
But when I launch the e-book viewer, the highlight does not re-appear.

I then refresh the Annotations browser, the highlight is gone.
I check the Annotations database table again, I see that calibre has updated annot_data in that record with the JSON of a deleted record.

I have deleted the OPF file of the book in the library folder after syncing the highlights and before re-opening calibre.

I have also disabled the option for keeping a copy of annotations/bookmarks in the e-pub file (which I found out about thanks to this thread).
I had this option previously enabled, so I edited the EPUB and I deleted the calibre.bookmarks.txt file inside it.

Despite all of this, calibre is keeping highlights data somewhere and restores it when the E-Book viewer is launched.
Where is calibre storing this data, and why does it bring it back?

Thanks in advance.

kovidgoyal · 01-14-2024, 11:10 AM

Highlights are stored in upto three different places.

1) The ebook file itself if it is EPUB format and the user has not disabled storing of highlights in the book file

2) A config file that keys stored highlights by absolute path to the book file

3) The database if the book comes from a calibre library

All these sources are merged by timestamp to present the final list of highlights in the viewer.

LuisNeto · 01-14-2024, 02:40 PM

Hi, Kovid. Thank you for your response.

I was already removing the annotations from those 3 places, but they kept being restored.

And I have just found out the 4th place where JSON files with the annotations were being stored: ~/Library/Preferences/calibre/viewer/annots.
(I forgot to mention in my original post that I am on macOS).

By deleting those files, in addition to the database records and the book's OPF file, and then syncing the highlights again, it works now: the E-book Viewer is no longer restoring the old state and it's displaying all the highlights!

Out of curiosity, may I ask what is the purpose of keeping the annotations in this folder as well, in addition to the 3 other places?

kovidgoyal · 01-14-2024, 09:50 PM

That is the config file I was referring to. The name of the json file is hash of the absolute file path, size, mtime, etc.

01-14-2024, 10:55 AM	#1
LuisNeto Member Posts: 10 Karma: 10 Join Date: May 2022 Device: Kobo Libra 2	Launching E-book viewer restores deleted annotations / highlights I am working on the development of a program for importing highlights from Kobo e-readers into calibre, and I've encountered a problem due to how calibre manages deleted annotations. This program works by inserting the highlights into calibre's Annotations database table. When I open calibre, I can see the highlights in "Browse annotations" and the E-book viewer. However, I start getting problems after deleting highlights in calibre. When a highlight is deleted, calibre updates the value of annot_data with an indication that it has been removed. When I run the program to import the highlights again, it gets this database record, verifies that its JSON in annot_data (which says it's removed) is different from the JSON containing the highlight data from the e-reader, so it updates the database record with the highlight data again, overwriting the JSON that calibre had written (which indicated it was removed). When I launch calibre, if I click in "Browse annotations" I can see the highlight. But when I launch the e-book viewer, the highlight does not re-appear. I then refresh the Annotations browser, the highlight is gone. I check the Annotations database table again, I see that calibre has updated annot_data in that record with the JSON of a deleted record. I have deleted the OPF file of the book in the library folder after syncing the highlights and before re-opening calibre. I have also disabled the option for keeping a copy of annotations/bookmarks in the e-pub file (which I found out about thanks to this thread). I had this option previously enabled, so I edited the EPUB and I deleted the calibre.bookmarks.txt file inside it. Despite all of this, calibre is keeping highlights data somewhere and restores it when the E-Book viewer is launched. Where is calibre storing this data, and why does it bring it back? Thanks in advance.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Kobo book management site (highlights/progress/annotations)	rigogsilva	Kobo Reader	28	09-01-2022 01:42 PM
Calibre e-book viewer and annotations	sdspieg	Calibre	2	04-28-2018 10:30 AM
Viewer with Annotations and Highlights	Cjammi	ePub	0	03-07-2017 04:07 AM
How do I update book info on device (Sony) without losing annotations & highlights?	Eugenie	Devices	3	01-11-2012 02:16 AM
PRS-600 Restore deleted annotations and highlights	facruz	Sony Reader	0	01-29-2011 11:10 PM

01-14-2024, 11:10 AM	#2
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Highlights are stored in upto three different places. 1) The ebook file itself if it is EPUB format and the user has not disabled storing of highlights in the book file 2) A config file that keys stored highlights by absolute path to the book file 3) The database if the book comes from a calibre library All these sources are merged by timestamp to present the final list of highlights in the viewer.

01-14-2024, 02:40 PM	#3
LuisNeto Member Posts: 10 Karma: 10 Join Date: May 2022 Device: Kobo Libra 2	Hi, Kovid. Thank you for your response. I was already removing the annotations from those 3 places, but they kept being restored. And I have just found out the 4th place where JSON files with the annotations were being stored: ~/Library/Preferences/calibre/viewer/annots. (I forgot to mention in my original post that I am on macOS). By deleting those files, in addition to the database records and the book's OPF file, and then syncing the highlights again, it works now: the E-book Viewer is no longer restoring the old state and it's displaying all the highlights! Out of curiosity, may I ask what is the purpose of keeping the annotations in this folder as well, in addition to the 3 other places?

01-14-2024, 09:50 PM	#4
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	That is the config file I was referring to. The name of the json file is hash of the absolute file path, size, mtime, etc.