[GUI Plugin] Highlights to Obsidian

[petzi]

Hi,

Whenever I click at "Send all highlights of selected books", I will get several files. For example, for the book I am currently reading, I will get 6 different files. One file (with the latest highlights) gets the header, the other 5 files are highlights from previous highlighting working sessions. They are not clear-cut per day, so I assume they have to do with my previous (test) exports with your plugins.

The same happens when I use "Resend previously sent highlights". I get for every book numerous files. It works correctly if I change for my last highlighted book the date of the last highlights sent to the day before I did some highlighting.

Hopefully, this description of the wrong behavior gives you some clue where to look…

Other issues:

• There is no automatic reminder in calibre that the plugin has a new version, as it is the case with other plugins.
• It would be nice to have the version number of the plugin in a handy place (perhaps in the config window?)
• You are right that (most) of the emojis work. But unfortunately, I have tested it with some they didn't work and still do not work, like ✏️ or 🙋🏻.
• Whenever I open the configuration window, I cannot configure something and then send highlights. Cancel, OK of the small configuration window (Formating Options / Other Options) has the same effect: it closes everything, even the big window in the background. But this wrong behavior happens only when I have opened the "Formatting Options" or "Other Options" window immediate previously.

BTW: I have written an article about your plugin in my German blog: Calibre E-Books Workflow – Von Calibre zu Obsidian

[jm289765]

Originally Posted by petzi

You are right that (most) of the emojis work. But unfortunately, I have tested it with some they didn't work and still do not work, like ✏️ or 🙋🏻.

Both of these work for me when I copy/paste them into the formatting options. The problem might be device-specific or related to your input method.

It's worth noting that both of these emojis are more than one character long. After backspacing 🙋🏻 once, I'm left with 🙋. And ✏️ turns into ✏ (in Obsidian, ✏ looks the same as ✏️). Copy-pasting single-character versions of emojis might work for you.

Originally Posted by petzi

Whenever I open the configuration window, I cannot configure something and then send highlights. Cancel, OK of the small configuration window (Formating Options / Other Options) has the same effect: it closes everything, even the big window in the background. But this wrong behavior happens only when I have opened the "Formatting Options" or "Other Options" window immediate previously.

If I'm reading this right, you're doing this:

1. Open config menu
2. Open "Formatting" or "Other" submenu
3. Close submenu
4. Open submenu again
5. While the submenu is still open, click "OK" or "Cancel" on the main config menu
6. After closing the config menu, the submenu closes itself

The fifth step isn't supposed to be possible; the submenu is a modal and should block interaction with the other windows. If you can still interact with the config menu, it's probably another device-specific thing or possibly an old version of calibre. I think I know what's causing the problem you described, but I won't be able to test, so maybe it'll be fixed, maybe not.

Originally Posted by petzi

Whenever I click at "Send all highlights of selected books", I will get several files.

I made it do this in 1.1.1 because there were problems with URI length. Since that's another thing that's probably device-specific, I'll add an option to change it.

The problem could also be something like this:

Originally Posted by jm289765

[...] with the text á͈̗̙̕ͅb̝̘͍̽͌̄͝c̷̝̣̭̣̯͉̺͔̋ͯ͐̈. It worked, but when it was in the title of the note sent to Obsidian, it always made a new note instead of adding to the existing one. This is probably an Obsidian problem, not something I can fix.

Originally Posted by petzi

There is no automatic reminder in calibre that the plugin has a new version, as it is the case with other plugins.

Should be fixed after my next post. I'll have made 10 posts and will be able to update the file attachment in the OP.

[jm289765]

Version 1.3.1

• Added options to customize maximum note size
• Current H2O version is now shown in main menu and config menu
• Rearranged drop-down menu to match main menu's button order
• Possibly fixed a possible bug where closing the main config menu would also close any open config submenu

[petzi]

1. Emojis

Copy-pasting single-character versions of emojis might work for you.

You are right. The emoji problem is only with emojis more than one character long.

2. Automatic reminder of new version

Should be fixed after my next post.

Yes, it worked! That is now very comfortable. Also, to see the version number is very helpful.

3. Config menu

If I'm reading this right, you're doing this:

1. Open config menu
2. Open "Formatting" or "Other" submenu
3. Close submenu
4. Open submenu again
5. While the submenu is still open, click "OK" or "Cancel" on the main config menu
6. After closing the config menu, the submenu closes itself

It works your way if I am opening the menu via the plugin button in Calibre. But I am referring to opening the config menu from the option menu of your plugin. See https://imgur.com/amFFRrR

When I open the "Formatting" or "Other" submenu in the setting of the picture above, I expect that – after closing the subwindow with the "Formatting" or "Other" submenu – that the main option window will remain open.

4. Export behavior: Double entries

The most pressing issue: I got every highlight and note twice.

Here is my note body format:

[Highlighted]({url}) on {date} at {time} UTC {timeoffset}:
{blockquote}

**Note:**
{notes}

---

And here is my body format for highlights without notes:

[Highlighted]({url}) on {date} at {time} UTC {timeoffset}:
{blockquote}

---

5. Export behavior: Splitting files

I understand now why my export produced several files. But getting several files is not very useful (at least for me). I started to highlight a new book with just a few highlights so that I am under your proposed limit of 20,000 characters. This limit is reached easily – I am working with 300,000 characters (science books of 700 or more pages).

With small files, the separations with a new header worked. But with the book, I am currently reading, I have about 150,000 characters (technically 75,000, but each entry is exported twice). With this book, I had problems with the header, even when I split the file. My header uses the Templater plugin and in some (not all!) separated files – as well as in the big file without separation – the export did not replace the variables in the header.

[jm289765]

Originally Posted by petzi

When I open the "Formatting" or "Other" submenu in the setting of the picture above, I expect that – after closing the subwindow with the "Formatting" or "Other" submenu – that the main option window will remain open.

I don't know what causes this. Is there another plugin you use that has a similar menu arrangement but doesn't have this problem? Maybe I can look at its source code to find a solution.

Originally Posted by petzi

The most pressing issue: I got every highlight and note twice.

Does this happen for every book or only some? Does it happen both for highlights with notes and highlights without notes? Does it happen for headers?

If you have a header, H, and two highlights, A and B, which order do they appear in: H A A B B, or H A B H A B, or something else?

Originally Posted by petzi

With small files, the separations with a new header worked. But with the book, I am currently reading, I have about 150,000 characters (technically 75,000, but each entry is exported twice). With this book, I had problems with the header, even when I split the file. My header uses the Templater plugin and in some (not all!) separated files – as well as in the big file without separation – the export did not replace the variables in the header.

Were none of the variables in the header replaced, or are there some that were replaced and some that weren't?

[petzi]

1. Submenu:
I will have a look and come back if I find a similar menu configuration.

2. Double highlights
I got every highlight twice. It doesn't matter whether it has included a note or not. And it happens with every book. The order is H A A B B.

3. Variables not instantiated in some headers
Whenever the header is not correct, then all the variables are not replaced by their values.

[petzi]

Examples of the submenu problem are the plugins "View Manager" and "GoodReads". Clicking on the button opens a dropdown menu under the button. Similar with your plugin button if clicking into the area where the down arrow is situated.

See picture 1-3 of https://imgur.com/a/OFwjVtO

Only when users choose the configuration menu from the dropdown menu, it opens another window where you can change values. Whenever this window is open, you can open another window. An example you can see in picture 4 of https://imgur.com/a/OFwjVtO: Clicking on the "Edit 'Shelf Add' Action…" button in the Goodreads configuration window. Closing the "Edit 'Shelf Add' Action…" window closes only this window.

-----

A solution for your plugin could be to open just a window for configuration issues. Maybe you could also spare the submenu "Formatting" and "Other" by putting the "Other" choice as a button into the formatting option window and to open an overlaid window for these choices. (Similar as "View Manager" and "GoodReads" do this with their other buttons inside the main configuration window.)

[jm289765]

Version 1.3.2
- Fixed error when trying to send a highlight that contains curly brackets
- Possibly fixed a bug where the main config window would close when closing a submenu
- Possibly fixed issues with highlights being sent twice and headers not being formatted
- Changed single-number time formatting options to give 2 or 4 digit results. e.g. {minute} will now give "05" instead of "5".

Originally Posted by petzi

1. Submenu
[...]
2. Double highlights
[...]
3. Variables not instantiated in some headers

Since I haven't been able to reproduce these problems on my computer, I don't know for sure that they're fixed. Let me know if any of them continue to happen.

[petzi]

Did you accidentally upload the old version? Or did you forget to update the version number? I cannot see any difference, and the version number is still 1.3.1.

I downloaded the attached file in the post above, as there was no automatic announcement for a new version. Maybe this happened because of the short time difference between uploading the new version and indexing for announcement?

[petzi]

I received the announcement of your plugin update and installed it via the standard Calibre feature. Thank you!

But I am sorry, I could not see any difference in the behavior:

1. The main menu is not a modal window. I see the difference with other plugins. When I use the "GoodReads" or the "View Manager" config window, I can click with the mouse beside the window without any effect. In contrast, your main window disappears behind the front calibre window. (But not the config menu. The config menu is correctly a modal dialog window.)

Besides solving this problem of the modal window, another solution could be to remove the choice "Configure this plugin" from the main menu. Then people have to select this choice via your H2O button and everything works fine.

2. I still get all the highlights twice.

3. The variables of the variables of the YAML header are not replaced by their values for the big file (477,000 characters). But it worked with a smaller file (151,000 characters).

4. Despite I limited the big file with 150,000 characters, I noticed that the plugin exported all the content, all 477,000 characters.

[musstank]

"ValueError:startfile: filepath too long for Windows"
I've got this error. Don't even understand what it means. I've managed to send highlights from one book, but I can't do it for all books and for one specific (in both cases this error appears).
Don't think that problem is with the file name, because I tried different variants, for example just "1". That book from which I successfully extracted highlights (just a second ago), had small amount of them, and that which gives an error, has many of them.

Anyway, great plugin idea, just found out about it!

Full error message:

calibre 6.14 embedded-python: True
Windows-10-10.0.18363-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18363')
Python 3.10.1
Windows: ('10', '10.0.18363', 'SP0', 'Multiprocessor Free')
Interface language: ru
Successfully initialized third party plugins: Extract ISBN (1, 6, 0) && Highlights to Obsidian (1, 3, 2)
Traceback (most recent call last):
File "calibre_plugins.highlights_to_obsidian.button_act ions", line 144, in send_new_selected_highlights
File "calibre_plugins.highlights_to_obsidian.button_act ions", line 55, in send_highlights
File "calibre_plugins.highlights_to_obsidian.highlight_ sender", line 667, in send
File "calibre_plugins.highlights_to_obsidian.highlight_ sender", line 31, in send_item_to_obsidian
File "webbrowser.py", line 86, in open
File "webbrowser.py", line 607, in open
ValueError: startfile: filepath too long for Windows

[musstank]

Originally Posted by petzi

2. I still get all the highlights twice.

I also have them twice, and I think it might be related to that they are also doubled in View -> Browse Annotations.
I can trace how they get duplicated for me.
First, I make highlights in Content-Server. They are labeled in "Browse Annotations" with the authorship of <username> (login in browser). Then, if I open the same book in Calibre Desktop app, all annotations get duplicated with the authorship of "Local E-book viewer user".
So, maybe fixing this should have something to do with choosing only one highlights "author". (If the genealogy of duplicating for others is the same as for me. And if it shouldn't be fixed in Calibre itself, not the plugin...)

[jm289765]

Originally Posted by musstank

"ValueError:startfile: filepath too long for Windows"

Go to config -> Other Options -> Maximum Note Size. Make sure "Restrict length of sent notes [...]" is checked. Gradually reduce the maximum size until this error goes away. After sending highlights of the affected book, you can reset the max size.

Originally Posted by petzi

1. The main menu is not a modal window.

I'm keeping this non-modal. It's convenient to be able to interact with the main window while the H2O menu is open. And the H2O menu doesn't have any functionality that would require blocking interaction with the main window.

Originally Posted by petzi

Besides solving this problem of the modal window, another solution could be to remove the choice "Configure this plugin" from the main menu. Then people have to select this choice via your H2O button and everything works fine.

IMO, the convenience of easy access to the config menu outweighs the invonvenience of having said config menu close when you exit a submenu. So I'm leaving that as it is.

The config menu bug won't be fixed; too time-consuming for something relatively minor.

Originally Posted by musstank

First, I make highlights in Content-Server. They are labeled in "Browse Annotations" with the authorship of <username> (login in browser). Then, if I open the same book in Calibre Desktop app, all annotations get duplicated with the authorship of "Local E-book viewer user".

This is useful information. I need to specify the restrict_to_user parameter when retrieving annotations.

[jm289765]

Originally Posted by petzi

2. I still get all the highlights twice.

3. The variables of the variables of the YAML header are not replaced by their values for the big file (477,000 characters). But it worked with a smaller file (151,000 characters).

4. Despite I limited the big file with 150,000 characters, I noticed that the plugin exported all the content, all 477,000 characters.

Does the attached file fix any of these?

I adjusted it to only use highlights from the local ebook reader, which should fix #2 (I might add a config option to toggle between local ebook reader and web).

I also simplified the code for formatting and sorting highlights, which might fix #3 and #4.

[petzi]

Sorry for not getting back to you sooner. I was out of town.

• Number 2 (all highlights twice) is fixed!
• Number 3 (Not replacing YAML variables) worked as before: Worked for smaller files (38,000 characters) but failed with longer files (238,687 characters).
• Number 4: (didn't limit file size) is fixed! It generated several files according to the limit set in the config options.

I experimented with different file limits to get more data for number 3 with a big file (238,687):
Maximums file size 30,000: 9 files, file 2 and 5 did not replace variables. But with another trial, only 1 and 9 replaced variables. Ye another trial: 1,3, 4 and 7 replaced variables. There appears to be no regularity.
Maximum file size 50,000: 5 files, several trials, always the second file does not replace the variables. There is regularity!
Maximum file size 80,000: 4 files, all files ok. Numerous trials.
Maximum file size 100,000: 3 files, only number 3 replaced variables in various trials. There is regularity.

I tested the option "Restricted length" but not to include headers in smaller notes. I noticed that all parts of the notes files are without a header. This surprised me: I thought that the first file should have the full header, all the following smaller files will not have a header.