Glo Annotations support and export

[Lynx-lynx]

Originally Posted by peitsao

You can copy and paste the notes in calibre's comment field to a document, and I noticed that Kobo's android app will let you send your highlights, etc. to other programs for saving.

I copy and paste annotations that I've made in the Calibre version of books to Citavi (free version), which I use as my research database.

Calibre's great, eh!!

[Lynx-lynx]

Hang about, I've just found the Annotations, and highlighting, in the section on the far right of the Calibre main library screen that shows the book you've highlighted, and the title, format and metadata info.

But I haven't found it at the back of the Calibre version of the book.

Golly, if I end up with say 100 annotations or notes, and even more highlights - not an unnusual proposition in non-fiction research, then surely they don't all sit there!!! Yikes ...

I suppose this really is a question for the Calibre forum, eh

[Shardancalibre]

Originally Posted by Lynx-lynx

Hang about, I've just found the Annotations, and highlighting, in the section on the far right of the Calibre main library screen that shows the book you've highlighted, and the title, format and metadata info.

But I haven't found it at the back of the Calibre version of the book.

Golly, if I end up with say 100 annotations or notes, and even more highlights - not an unnusual proposition in non-fiction research, then surely they don't all sit there!!! Yikes ...

I suppose this really is a question for the Calibre forum, eh

I don't have the kobo here before my eyes (I'll have it in a week or so I suppose) so I practically can't imagine the process. Tell me if I'm right:

You take ur notes and higlights on ur device. Then you use a command (fetch annotations) within calibre that's supposed to collect ur annotations someplace so that you can copy them and export them?

Is that correct?

[davidfor]

Originally Posted by Shardancalibre

I don't have the kobo here before my eyes (I'll have it in a week or so I suppose) so I practically can't imagine the process. Tell me if I'm right:

You take ur notes and higlights on ur device. Then you use a command (fetch annotations) within calibre that's supposed to collect ur annotations someplace so that you can copy them and export them?

Is that correct?

Basically yes. Calibre copies the annotations into the comments field in its metadata for the book. If you want to put them somewhere else, you have to cut-and-paste from there.

[Shardancalibre]

Originally Posted by davidfor

Basically yes. Calibre copies the annotations into the comments field in its metadata for the book. If you want to put them somewhere else, you have to cut-and-paste from there.

Do you happen to know if it can do the same with a kindle device? Just to see practically how this method works (At least my old kindle could prove itself useful after it got its screen broken).
Because if I can take the annotations from calibre, and they are not the mess I've seen in that link I posted before, I would have solved my problems.

[davidfor]

Originally Posted by Shardancalibre

Do you happen to know if it can do the same with a kindle device? Just to see practically how this method works (At least my old kindle could prove itself useful after it got its screen broken).
Because if I can take the annotations from calibre, and they are not the mess I've seen in that link I posted before, I would have solved my problems.

I have never used a Kindle with calibre, so I don't know. For the Kobo, the menu option is on the send to device menu. If the Kindle supports this, it will be there.

[Shardancalibre]

Originally Posted by davidfor

I have never used a Kindle with calibre, so I don't know. For the Kobo, the menu option is on the send to device menu. If the Kindle supports this, it will be there.

Ok thanks. I tried with the kindle but I didn't see anything that could remind that command. I'm sure I'll be needing your help when I get the Kobo. Bye for now.

edit: I've found the command (it says experimental). In a nutshell when I send a book to my device I use the option "fetch annotations". Then when I finish the reading and have taken my annotations, I connect my device to calibre again, and I should find all of them in the metadata field, am I right?

edit2: failed experiment. All the annotations it took were the positions of the bookmarks and the percentage of the book I had read. I hope with the kobo will work better.

[franklekens]

Originally Posted by davidfor

Basically yes. Calibre copies the annotations into the comments field in its metadata for the book. If you want to put them somewhere else, you have to cut-and-paste from there.

I think the experimental "fetch annotations" function isn't functioning entirely correctly yet. Just tried it, and for one book it only put *part* of the annotations in the comment field.


Incidentally, on the reader itself (Aura), the annotations and bookmarks are often displayed in a weird order. E.g. a bookmark placed on page 30 will sometimes sit before or between highlights from page 20 in the list the Aura displays to me. Really rather annoying.

Is there an easy way to read and edit the xml files, or extract just the annotations from it? I'm not used to editing xml and all the xml codes are kind of distracting. I'd like an easy way just to get the highlighted texts and annotations in a simpler word processor.

[davidfor]

Originally Posted by franklekens

I think the experimental "fetch annotations" function isn't functioning entirely correctly yet. Just tried it, and for one book it only put *part* of the annotations in the comment field.

I haven't used fetch annotations recently, but I can't say I have ever seen any problems with it.

Incidentally, on the reader itself (Aura), the annotations and bookmarks are often displayed in a weird order. E.g. a bookmark placed on page 30 will sometimes sit before or between highlights from page 20 in the list the Aura displays to me. Really rather annoying.

There are some bugs with the annotation management. This is one of them.

Is there an easy way to read and edit the xml files, or extract just the annotations from it? I'm not used to editing xml and all the xml codes are kind of distracting. I'd like an easy way just to get the highlighted texts and annotations in a simpler word processor.

Not without writing some code.

[franklekens]

Originally Posted by davidfor

I haven't used fetch annotations recently, but I can't say I have ever seen any problems with it.

There are some bugs with the annotation management. This is one of them.

Is there an easy way to read and edit the xml files, or extract just the annotations from it? I'm not used to editing xml and all the xml codes are kind of distracting. I'd like an easy way just to get the highlighted texts and annotations in a simpler word processor.[/QUOTE]

Not without writing some code.[/QUOTE]


That's what I suspected. A pity no one has written a simple editor or extraction programme yet that convert the annot-files into something more easily usable for the average reader.

It's beyond my abilities. I've just had a look at the xml in Editpad. I thought basically deleting all text between </text> and <text> would serve my purposes -- but my knowledge of regular expressions is so limited I can't even manage that (i.e. remove each string of </text><anything in between><text>).

By the way, I'm pretty sure the fetch annotation command left me with only part of the highlights for one book in the comments field. I compared it with the annot file, and the highlights missing in Calibre really were there in the file.

[davidfor]

Originally Posted by franklekens

That's what I suspected. A pity no one has written a simple editor or extraction programme yet that convert the annot-files into something more easily usable for the average reader.

It's beyond my abilities. I've just had a look at the xml in Editpad. I thought basically deleting all text between </text> and <text> would serve my purposes -- but my knowledge of regular expressions is so limited I can't even manage that (i.e. remove each string of </text><anything in between><text>).

By the way, I'm pretty sure the fetch annotation command left me with only part of the highlights for one book in the comments field. I compared it with the annot file, and the highlights missing in Calibre really were there in the file.

It is possible there was an inconsistency between the annot file and what is in the devices database. Fetch annotations is reading the database, not the files. There are bugs related to this, but, I didn't think they caused any data loss.

There are some other ways to get the annotations off the device. My utilities plugin will get the annotations for selected books as text. But, it uses the same code as the fetch annotations. Someone, tshering I think, has written a little utility to fetch them. And I'm pretty sure someone has posted a script to do it as well.

[eschwartz]

For the Kindle, I would just use the Annotations plugin, it works very well. Someone should add compatibility with the Kobo .xml's.

[tshering]

Originally Posted by davidfor

Someone, tshering I think, has written a little utility to fetch them. And I'm pretty sure someone has posted a script to do it as well.

Since the tool (for windows) is at a somewhat hidden place, here is the link. I only shortly tried the export feature of the new firmware. Therefore, I don't know how the results compare, but I am certain that my tool is more user friendly. (As for information: The tool offers an option to try to repair the database. Since this was written with respect to a specific FW, I would not recommend to use this feature any longer.)

[davidfor]

Originally Posted by eschwartz

For the Kindle, I would just use the Annotations plugin, it works very well. Someone should add compatibility with the Kobo .xml's.

I had a look a while ago and did about half of it. But, the plugin doesn't use the ondevice hints in the library. It does its own matching and it was 100% when I tested it. I haven't looked at it recently, so it might have changed.

[pakoe]

This script extracts all texts (highlighted texts, annotation texts, notes added to bookmarks) from the annotation xml files in folder /mnt/onboard/Digital Editions/Annotations. It is meant to run on the Kobo itself. The resulting file /mnt/onboard/pakoe/_annot.html is sorted and formatted. By adding on your Kobo a certain string to an annotation you can hide texts older than that annotation (useful if you are working through your annotations chronologically). _annot.html must already exist before (just create an empty file). The script could be started using Kobo Tweaks, I start it by simply opening _annot.html (open it again to refresh) having used fmon from KoboLauncher to make _annot.html an "alias" for the script file _annot.sh by the following line in /mnt/onboard/.kobo/on_start.sh

/mnt/onboard/pakoe/fmon /mnt/onboard/pakoe/_annot.html /mnt/onboard/pakoe/_annot.sh 2>&1 &

If you neither want to use Kobo Tweaks nor fmon, call _annot.sh from /mnt/onboard/.kobo/on_start.sh. Then _annot.html will be refreshed only by restarting the Kobo.
Of course _annot.html can't be used to edit or delete annotations or to open a book and jump to an annotation. _annot.html just shows ALL annotations (or all newer than one you selected) from ALL books well formatted and sorted. And you can view and refresh it on the Kobo without connecting to a computer.
If you have sqlite3 on your Kobo (it is included for example in "Kobo Start Menu", see https://www.mobileread.com/forums/sho...d.php?t=233259), a much simpler solution is possible, reading the annotations from the file .kobo/KoboReader.sqlite on the Kobo.
So here is _annot.sh (probably it still could be simplified):

filePath=/mnt/onboard/pakoe/
fileName=$filePath"_annot.html"
tmpFile=$filePath"tempAnnot"
annotLog=$filePath"_annot.log"
hideIfOlderThanThis="kkqqq"
count=1
date > $annotLog

mySed()
{
	echo "$1" >> $annotLog
	sed -r "$1" $fileName >> $tmpFile 2>> $annotLog
	#fileName=$filePath"tmp"$count # Auf die Datei $fileName werden bis zum nächsten mySed evtl. noch weitere Befehle angewandt, erst dann wird fileName neu definiert. Die Datei $fileName hat also nicht den Zustand unmittelbar NACH einem mySed, sondern den unmittelbar VOR einem mySed (bzw. am Ende des Scripts)!
	count=`expr $count + 1`
	mv $tmpFile $fileName
}
mySort() # setzt voraus, dass alle \n durch # ersetzt sind, ersetzt aber das letzte # vor jeder Annotation  wieder durch \n (dadurch enthält jede Zeile genau 1 Annotation) (die erste Zeile theoretisch vielleicht gar keine)
{
	mySed 's!#([^#]*<annotation>)!\n\1!g'
	# Copy something to the beginning of each line to determine sorting order, mark its ending by =:
	mySed $1
	sort -f $fileName -o $fileName
	# Remove again what was copied to determine sorting order:
	mySed 's!^[^=]+=!!g'
}

mv $fileName $filePath"_annot.bak.html"
# "^ *[^ <]" looks for continuation lines of multi-line texts. Because of the number of spaces before <dc:date> not the <dc:date> tag inside the <content> tag of an <annotation> tag is found (change date), but the <dc:date> tag inside the <annotation> tag itself (creation date). So if you are working through your annotations chronologically, you can hide older annotations by adding the value of hideIfOlderThanThis to a (new or existing) annotation:
grep -E '<annotation>|<text>|<content>|^ *[^ <]|^    <dc:date>' -r "/mnt/onboard/Digital Editions/Annotations" > $fileName 2>> $annotLog
# _<<1, _<<2 can safely be used as markers, as any < IN A TEXT has been replaced by &lt; in the .annot file (which of course contains many <, but no <<):
mySed 's!#!_<<1!g'
mySed 's!=!_<<2!g'
tr '\n' '#' < $fileName >> $tmpFile 2>> $annotLog
mv $tmpFile $fileName
mySort 's!^.*<dc:date>20(..-..-.....:..:..)!\1=\0!g'
# Remove annotations without text (bookmarks):
mySed 's!^!=!g'
mySed 's!^=(.*<text>)!\1!g'
mySed 's!^=.*$!!g' # Falls etwas gefunden wird, entsteht eine Leerzeile, d.h. die Zeile wird nicht gelöscht
tr '\n' '#' < $fileName >> $tmpFile 2>> $annotLog
mv $tmpFile $fileName
# alles vom Anfang greedy bis zum Anfang der letzten Zeile, die $hideIfOlderThanThis enthält, löschen:
mySed 's!.*#(.*?<annotation>.*?'$hideIfOlderThanThis')!\1!g'
mySort 's!^([^:]*)/([^:/]+):.*<dc:date>20(..-..-.....:..:..)!\2#\1#\3=\0!g'
mySed 's!#!\n!g'
mySed 's!^[^:]*/([^:/]+\.annot:)!=\1j=\1!g'
tr '\n' '#' < $fileName >> $tmpFile 2>> $annotLog
mv $tmpFile $fileName
# Now # is a line break, = and j= mark the beginning of an annotation file name.
# Leerzeilen wurden beim Sortieren nur durch # dargestellt und daher NICHT aus dem Weg geräumt und müssen hier berücksichtigt werden:
mySed 's!j(=[^:]+\.annot:)([^#]*#+)\1!\2!g'
mySed 's!#!\n!g'
mySed 's!j=[^:]+\.annot:!!g'
mySed 's!=([^:]+)\.annot:!<br /><b><i><u>\1</u></i></b>!g'
mySed 's!<text>!!g'
mySed 's#</text>##g'
mySed 's!<content>!<b>#</b>!g'
mySed 's#<annotation>#<br />#g'
mySed 's!'$hideIfOlderThanThis'!<b><i>\0</i></b>!g'
mySed 's#<dc:date>.*(..)[^0-9]0?([1-9][0-9]?)[^0-9]0?([1-9][0-9]?)[^0-9]+(..:..):.+</dc:date>#<b>\3.\2</b>.\1,\4#g'
mySed 's!_<<1!#!g'
mySed 's!_<<2!=!g'
echo '<html><head>' > $tmpFile
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />' >> $tmpFile
echo '</head><body>' >> $tmpFile
date +"%-d.%-m.%y %-H:%M" >> $tmpFile
echo ' ("<b><i>'$hideIfOlderThanThis'</i></b>" in einer Annotation blendet alle &auml;lteren aus!)' >> $tmpFile
cat $fileName >> $tmpFile
echo '<br />-------------------END-------------------' >> $tmpFile
echo '</body></html>' >> $tmpFile
mv $tmpFile $fileName