[GUI Plugin] Annotations

[Atcold]

Originally Posted by davidfor

Yes, calibre version 5 uses Python 3. There are differences in the language and the plugins needed to be updated to handle this. A lot is backwardly compatible with Python 2 but some things need to be handled for each version. At the moment, I am maintaining compatibility with older calibre versions of calibre. I broke the compatibility with a change, and then broke everything fixing that.

We're talking about the buggy sorting, right? Can you expand a little (or link the diff)?

So, in the attached beta I have change the way to get the list of books. […] It should be accurate as it uses a calibre search to get the books on the device that are in the library and then uses other calibre functions to get the path to the book. This should work better than the existing method. Unfortunately, I have no way to test it. I don't have a Kindle or a good way to simulate it. The code should be good, but, there might be something. Please run it and send me the debug lot to look at.

Awesome! I cannot see anything, but it works!

Log: tmp848au2i5.txt.zip

[davidfor]

Originally Posted by Atcold

We're talking about the buggy sorting, right? Can you expand a little (or link the diff)?

In that particular case, I was talking about how the My Clippings.txt file is read. With the move to Python 3, strings, unicode, UTF etc are handled differently. Add that some My Clippings.txt files seem to have different formatting and at least one has unicode BOM characters in a location I wouldn't have expected them (but sort of make sense), I have been going back and forth on where and how to decode the file. I changed it to as part of the open of the file, but, that parameter is not part of the Python 2 file open. I put out a beta with that, it broke the plugin under calibre 4.x. When I fixed it, I only did half the job. It probably worked for calibre 4.x, but, for calibre 5.x, it would not have found read the My Clippings.txt file properly and would not have found any annotations. Which is part of what was happening for you.

I haven't checked these changes in. I generally only do that when I make the actual release. It's a bit lazy of me, but, it is the way I work. If you want to look at the code, it is in "ParseKindleMyClippingsTxt.py", method "FromFileName" and the start of "FromFileName"

Awesome! I cannot see anything, but it works!
Attachment 184991

That looks good. But, it also says that you have only three books on the device that match to books in the library (have the green tick next to them). And the other five books that have annotations don't show as matched. Is that correct?

If so, you can fix this in the device view. Right-click (or whatever the equivalent on a MAC is) on an unmatched book and choose the "Match book to library" option. On the dialog that comes up, search for the book, select it and press OK to update the list. When finished matching them, right-click the device button and choose "Update cached metadata on device". This will update the metadata.calibre file on the device and calibre will match the book to the library in the future.

Log: Attachment 184990

That's horrible. If you click the "Hide match status" it will use the default list colours and should be more readable. Someone did report this a few weeks ago and I said I wasn't interested. I thought the posted a version of the plugin that did something different with colour, but, it was removed before I had a chance to look at it. I don't know if it simply set colours that were better for dark-mode or did something else. I have to do something for colours in another plugin, so I might visit it here when I have done that.

[Atcold]

Originally Posted by davidfor

[…] I put out a beta with that, it broke the plugin under calibre 4.x. When I fixed it, I only did half the job. It probably worked for calibre 4.x, but, for calibre 5.x, it would not have found read the My Clippings.txt file properly and would not have found any annotations. Which is part of what was happening for you.

So, what about the sorting bit? Haha

I haven't checked these changes in.

Oh, I thought you were using git / GitHub.

That looks good. […] Is that correct?

Yes. The others are Amazon's Kindle books, and their annotations can already be exported through Amazon's system.
I was trying to dump the annotations for Atomic Habits only, which I've sideloaded, and therefore wasn't showing up on Amazon's system. (I wonder if it would have shown in case I would have sent the book to my Kindle via email rather than USB. I guess we'll never know.)

That's horrible.

I agree.

Someone did report this a few weeks ago and I said I wasn't interested.

I saw that. But I tried to nudge you again anyways.

I thought they posted a version of the plugin that did something different with colour, but, it was removed before I had a chance to look at it.

I can still see it. ↓

Originally Posted by davidfor

Sorry, it's not really a problem I am interested in fixing. It either means coming up with a colour combination that always works, adding code to use different colours depending on the styles, or code to configure the colour choices.

I don't know if it simply set colours that were better for dark-mode or did something else.

This is the default on my machine (which is set to work in a dark mode).

I have to do something for colours in another plugin, so I might visit it here when I have done that.

Nice. Let me know if I can help. (Btw, the website does not notify me about updates, so I'm not sure how to be responsive.)
Also, what other plugins are you involved in? I'm curious now

[davidfor]

Originally Posted by Atcold

So, what about the sorting bit? Haha

The answer to that involves a significant number of four letter words

Oh, I thought you were using git / GitHub.

The source is on Github. I'm just lazy and don't check it in until I am ready. Generally, it is just me, so I haven't bothered. The source is in the plugin, so people who want to can see the beta code.

Yes. The others are Amazon's Kindle books, and their annotations can already be exported through Amazon's system.
I was trying to dump the annotations for Atomic Habits only, which I've sideloaded, and therefore wasn't showing up on Amazon's system. (I wonder if it would have shown in case I would have sent the book to my Kindle via email rather than USB. I guess we'll never know.)

If you had emailed it, it probably would have worked. The issue was in what calibre calculated the file name to be vs what it actually was. Calibre could use a generate name for Windows, MACs and Linux as the exact path is different and length of the path affects the filename generation.

I agree.



I saw that. But I tried to nudge you again anyways.



I can still see it. ↓

For some reason, those screenshots didn't look as bad as yours did. But, I meant they posted a version of the plugin with some changes. I planned to look at what they did, but, they removed it before I got around to it.

This is the default on my machine (which is set to work in a dark mode).



Nice. Let me know if I can help. (Btw, the website does not notify me about updates, so I'm not sure how to be responsive.)
Also, what other plugins are you involved in? I'm curious now

Far to many The main one is the Kobo Utilities. And I'm involved with all the other Kobo related plugins, though someone else owns them. After that, most are plugins created by @kiwidude who is no longer here maintaining his plugins. Several people are doing this based on what they what they use. Count Pages and Goodreads Sync are probably the main ones, but, I have updated several others to keep them working. I inherited the Annotations plugin when the original developer retired. Plus I look after the KoboTouch driver in calibre and occasionally other changes when I see a bug or something I want changed and think Kovid will agree.

[dunhill]

I have managed to make the plugin work, with the following criteria. If the book is passed through Cable from Calibre the file is housed in documents and if in that book we make a note or highlight something the complement takes it. On the other hand, if we send a book to the kindle by mail, as is my case, it places it in the Documents / Downloads / Items01 folder and from there it does not record anything.

[davidfor]

Originally Posted by dunhill

I have managed to make the plugin work, with the following criteria. If the book is passed through Cable from Calibre the file is housed in documents and if in that book we make a note or highlight something the complement takes it. On the other hand, if we send a book to the kindle by mail, as is my case, it places it in the Documents / Downloads / Items01 folder and from there it does not record anything.

Is that with the latest beta? With that, if the book on the device is matched to a book in the library, it won't matter where it is on the device. To that, it any books in the "Documents" directory, but, doesn't look at any sub-directories. Calibre should be searching these directories looking for books and trying to match them to the library. Though, I don't have a Kindle, so I don't know for sure if this works.

A complication to this is that the title and authors for the entries in the "My Clippings.txt" file are from the metadata in the books. When the plugin tries to match to the books, it is using these details. That could give some errors.

If the above doesn't explain what you are seeing, I need to see a debug log. Turn the debug option on in the plugin's configuration and run calibre in debug mode. Post/send the log and the "My Clippings.txt" file with some details of what is missing. Or you think is wrong.

[dunhill]

Originally Posted by davidfor

Is that with the latest beta? With that, if the book on the device is matched to a book in the library, it won't matter where it is on the device. To that, it any books in the "Documents" directory, but, doesn't look at any sub-directories. Calibre should be searching these directories looking for books and trying to match them to the library. Though, I don't have a Kindle, so I don't know for sure if this works.

A complication to this is that the title and authors for the entries in the "My Clippings.txt" file are from the metadata in the books. When the plugin tries to match to the books, it is using these details. That could give some errors.

If the above doesn't explain what you are seeing, I need to see a debug log. Turn the debug option on in the plugin's configuration and run calibre in debug mode. Post/send the log and the "My Clippings.txt" file with some details of what is missing. Or you think is wrong.

Plugin version 1.14.9
Calibre 5.10.1
Here I send you the txt files, two of which are My Clippings.txt from where I could get the notes and another My notes.txt where the notes from another book marked on the kindle are.
The kindle is with Spanish language.
And in the image the books that Calibre recognizes in the device.

[davidfor]

Originally Posted by dunhill

Plugin version 1.14.9
Calibre 5.10.1
Here I send you the txt files, two of which are My Clippings.txt from where I could get the notes and another My notes.txt where the notes from another book marked on the kindle are.
The kindle is with Spanish language.
And in the image the books that Calibre recognizes in the device.

Looking at these files:

"My Clippings.txt"

That has annotations or one book titles "56" by "Jorge Lanata". In it, the start of each has "La" instead of either "Mi" or "Tu". You reported a problem a while ago and I added "La subrayado" for annotations, but, I did not know if the other two should be added. The attached beta has "La nota" for notes and "La marcador" for bookmarks. Your file doesn't have an example bookmark, but, it is a fairly safe guess.

A book with just a number for the title is fairly unusual. And it is a very short title. It is possible this is not being handled properly. Could you test another book?

"Mi notas.txt"

This has the same format as the "My Clippings.txt" file. Is this a copy you made or did the Kindle produced it for those annotations? I have had a report of a different name for Kindles set to German and the plugin looks or that. But, it is only processing one file, the first of 'My Clippings.txt' and 'Meine Clippings.txt' that is found. I can add "Mi notas.txt" to this, but, I need to know something about when it is used.

The debug log

Unfortunately, there isn't anything useful in it. It doesn't look like a device was connected or that the annotations were fetched. I need the debug log fro when the annotations are fetched to see what is happening.

The log does show a lot of other errors. It looks like you have a templates using columns "#estado" and "#gr10>2s" that do not exist. And a few plugins that probably haven't been updated for Python 3.

[dunhill]

Thank you for the return and your time. I will do the relevant tests and comment.

[chavoosh]

Hi everyone,

I just bought a Kindle Oasis and I am trying to test "Annotation" plugin. I sent a book to my Kindle via Calibre (USB), highlighted a text, and then tried to fetch the annotation but the Annotation plugin said: "No annotated books found on your device."

How can I resolve this issue?
P.S. I tried both AZW3 and MOBI formats

[davidfor]

Originally Posted by chavoosh

Hi everyone,

I just bought a Kindle Oasis and I am trying to test "Annotation" plugin. I sent a book to my Kindle via Calibre (USB), highlighted a text, and then tried to fetch the annotation but the Annotation plugin said: "No annotated books found on your device."

How can I resolve this issue?
P.S. I tried both AZW3 and MOBI formats

Is this with the released version? If not, try the beta posted above. The beta's include fixes for Kindles especially related to KFX books.

If you are using the beta, I need some details. A debug log from calibre with the debug option in the plugin turned on. Plus the "My Clippings.txt" from from the device. Either post them here or upload them and send me a PM with the link to the files.

[dunhill]

@davidfor
Apparently this latest beta solved the export problems, of the books added through the cable and through the mail, the marks and notes are added to the file My Clippings.txt.
My plugins are all updated to the latest version so I will check the errors that you tell me. Thank you very much for all your answers and I will see to complete the translation of the plugin into Spanish, which in my opinion is 50%.
I leave the debug file.

[tothemoon]

Muddling through with a Kindle Oasis but the beta from year's end works a dream!

I'm here to inquire if two things are possible with note formatting (excuse any stupidity):

1. I would like to get rid of all location markers in the annotations (I only want text and notes). This doesn't seem possible in the 'Modify Appearance' dialogue. Is it possible?

2. I'd like to prepend a text string before every 'Note' in the annotations? I played around with the CSS annotation element for Notes but couldn't manage it.

[davidfor]

Originally Posted by tothemoon

Muddling through with a Kindle Oasis but the beta from year's end works a dream!

I'm here to inquire if two things are possible with note formatting (excuse any stupidity):

1. I would like to get rid of all location markers in the annotations (I only want text and notes). This doesn't seem possible in the 'Modify Appearance' dialogue. Is it possible?

You can try putting "display:none;" in the CSS for this. I don't know if that will work or not.

Not including it is a problem as contains information that is used when updating the annotation.

2. I'd like to prepend a text string before every 'Note' in the annotations? I played around with the CSS annotation element for Notes but couldn't manage it.

Do you mean you want something before each annotation if it is a "note" type? Or a text to indicate where the actual note starts to differentiate it from the highlighted text? Neither is possible at the moment and would need development (someway to configure it and then the code to handle it).

If it is the latter, the CSS is used to make the note and highlight part of the annotations different. I think I'd find extra words to be intrusive and not actually make things clearer. Exactly how to format it (bold, italics, size, same line? Hanging indent?) would also need to be thought about. And how much to make configurable. Or maybe a template that can be configured somewhere. I'm not sure of the best way, or if it is a good idea overall.

[tothemoon]

Originally Posted by davidfor

You can try putting "display:none;" in the CSS for this. I don't know if that will work or not.

Not including it is a problem as contains information that is used when updating the annotation.


Do you mean you want something before each annotation if it is a "note" type? Or a text to indicate where the actual note starts to differentiate it from the highlighted text? Neither is possible at the moment and would need development (someway to configure it and then the code to handle it).

If it is the latter, the CSS is used to make the note and highlight part of the annotations different. I think I'd find extra words to be intrusive and not actually make things clearer. Exactly how to format it (bold, italics, size, same line? Hanging indent?) would also need to be thought about. And how much to make configurable. Or maybe a template that can be configured somewhere. I'm not sure of the best way, or if it is a good idea overall.

Originally Posted by davidfor

You can try putting "display:none;" in the CSS for this. I don't know if that will work or not.

Not including it is a problem as contains information that is used when updating the annotation.

Is a 'clean copy' mode a possibility? Where just the text content can be exported (clipboard or .txt or whatever) and not interfere with any of the location data.

I'm formatting the notes in another application, I suspect it's a pretty regular use case. Given that clipping.io is now a paid service, your is the only way to do this for free. So hopefully more users!

[Current workaround is to use a unique string in the 'Timestamp format' field, which allows you to Find All in a text editor and get rid of the location markers]

Originally Posted by davidfor

Do you mean you want something before each annotation if it is a "note" type? Or a text to indicate where the actual note starts to differentiate it from the highlighted text? Neither is possible at the moment and would need development (someway to configure it and then the code to handle it).

Any textual marker to mark the start of a note would help! Just like the 'Timestamp format' allows, since it would enable post processing through a text editor/macros.

To bring the formatting options inside the plugin does open up some questions, and would be a luxury if the above was possible.
The CSS gets stripped in a text editor is my problem, but I understand mine might be a fringe case.