Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Kobo Reader > Kobo Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 10-13-2022, 02:00 PM   #1
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Exporting annotations, bookmarks and reading progress to the cloud

As I've lost my annotations and reading progress too many times (mostly due to playing around with scripts, installing stuff, automatic import etc.), and I don't use Calibre, I've hacked together a small script that exports my annotations, bookmarks and reading progress to Markdown files.

Currently I'm using it via NickelMenu like this:

Code:
menu_item :main    :Sync Annotations    :nickel_wifi     :autoconnect
  chain_success                         :cmd_spawn       :quiet:/mnt/onboard/.adds/notes/notes.sh
  chain_success                         :dbg_toast       :Syncing Notes
I've also added a sqlite3 binary to extract all the information I need from the Kobo database.

If you want to use it, extract the zip file to /mnt/onboard/.adds/notes/ and adjust the directory you want your notes exported to.

Click image for larger version

Name:	Screenshot from 2022-10-13 19-57-01.png
Views:	766
Size:	116.5 KB
ID:	197132

Feedback welcome!

Changelog:

2022-10-13:
  • Initial release

2022-10-14:
  • Fix libsqlite.so.0 typo
  • Export file to /mnt/onboard/Exported Annotations/Notes.md
  • Add current date/time to the file
  • Only show not hidden bookmarks
  • Show chapter in highlights
  • Show chapter in dogear bookmarks

2022-10-15:
  • Combine highlights and dogears to a single list
  • Add created date to each highlight
  • Add an icon showing whether an entry is a dogear, note or highlight
  • Fix compatibility with epub highlights
  • Add note regarding export location when not using English as default language

2022-10-16:
  • Change document icon in order to support Android
  • Add current chapter name to current reads
  • Add newlines between headings and quotes (not required by Markdown, but by pandoc)

2022-10-17:
  • Order bookmarks by title, chapter, chapter progress and then date
  • Simplify upload filename to "notes-%Y-%m-%d.md"

2022-10-17:
  • Removed the dependency to KoboCloud
  • Removed the Nextcloud Curl command
  • Exported notes are now named with their current date (e.g. notes-2023-02-17.md)
  • Notes are rotated and 21 notes are kept (can be configured)

2023-02-17:
  • Remove KoboCloud dependency
  • Rename notes with their current date

2023-08-04:
  • Export book progress of PDFs as well

2023-10-16:
  • Fix a bug in reading progress that didn't list a book when it doesn't have any chapter titles (e.g. PDFs or automatically generated Epubs)

2024-03-26:
  • Simplify SQL query and don't exclude bookmarks if Epub is partially broken or has no chapter titles
Attached Files
File Type: zip notes-20221013.zip (590.6 KB, 548 views)
File Type: zip notes-20221014.zip (590.8 KB, 261 views)
File Type: zip notes-20221015.zip (591.1 KB, 254 views)
File Type: zip notes-20221016.zip (591.3 KB, 327 views)
File Type: zip notes-20221017.zip (591.3 KB, 321 views)
File Type: zip notes-20230217.zip (591.2 KB, 245 views)
File Type: zip notes-20230804.zip (591.2 KB, 246 views)
File Type: zip notes-20231016.zip (591.1 KB, 597 views)
File Type: zip notes-20240326.zip (593.1 KB, 406 views)

Last edited by qkqw; 03-26-2024 at 07:51 AM.
qkqw is offline   Reply With Quote
Old 10-14-2022, 05:30 AM   #2
mtxo
Junior Member
mtxo began at the beginning.
 
Posts: 6
Karma: 22
Join Date: Oct 2022
Device: Kobo Nia
Hello. I just register me to reply to this post.

Many thanks for this script, it works well for me. But I don't use it for store annotations in the cloud. I have modify a bit the script and put that it copies de results to the Kobo itself, and them I save it to my PC through the usb cable.

It is more powerful that the Kobo system, that lets to save the annotations of only one book at once!!

But there is a bug in the zip. The archive in the lib folder need to name it as libsqlite3.so.0 (see the .0 at the end), and no as libsqlite3.so like in the zip file. With this change, the script is working well now!!

Many thanks for it. Excuse my bad english. Greetings. Mtxo.
mtxo is offline   Reply With Quote
Advert
Old 10-14-2022, 09:34 AM   #3
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Thanks for your feedback! I've indeed fixed the libsqlite3.so.0 typo on my Kobo, but not on my system I've just uploaded a new version with some more features and improvements, e.g. showing the current chapter and exporting the notes to /mnt/onboard/Exported Annotations/Notes.md instead of a temporary file.

If you want to share your version or any other improvements, I'd be happy to integrate them!
qkqw is offline   Reply With Quote
Old 10-14-2022, 02:54 PM   #4
mtxo
Junior Member
mtxo began at the beginning.
 
Posts: 6
Karma: 22
Join Date: Oct 2022
Device: Kobo Nia
Many thanks for the improvements.

I not interested in the cloud part of the script, so as now it saves the markdown file in the kobo storage, it is so fine for me.

In the first script I was made this "improvement":

Code:
#$CURL -u "$USER" -H "$HEADER" -T "$TMP" "$URL/notes-$DATE.md"
cp "$TMP" /mnt/onboard/mynotes-$DATE.md
So: comment the line that make the cloud connection, and put a new line for save temp file to the kobo storage. So, nothing interesting

I'm not a coder, so I can't improve your code

I test the new version. The new code for save the markdown file to the kobo storage is ok. But, maybe it have a "bug" in the case of a non english environment. Mine is spanish, and the "Exported Annotations" dir is called "Anotaciones exportadas" in spanish. So maybe you need to put a note in the script to the user to adjust it to her language...

And the result markdown file is not "the same" in the two versions. In the previous version I have a more complete file, whit probably all the highlights of the kobo saved. In the new one, there are not highlights for some books. You can see them in the attached files (i renamed them to the .txt extension). The 1.txt file is made with the first version, and the 2.txt file with the second.

As you can see, in 1.txt there are highlights from these books:
- Autonomous, Annalee Newitz
- Informe sobre Biodiversidad (a lot of them)
- Instrucciones para salvar el mundo, Rosa Montero
- La libertad interminable, Joe Haldeman
- Panteón, Carlos Sisí

In the second file there are only highlights from "Informe sobre Biodiversidad". There are all of them, but in another order. Maybe, the order of these highlights in the first file is better, but I not sure.

If you need my kobo database, I can send you, but I prefer to not make it public, so I will send to you only.

That's all. Many thanks for you work. Greetings. Mtxo.
Attached Files
File Type: txt 1.txt (5.2 KB, 299 views)
File Type: txt 2.txt (5.9 KB, 250 views)
mtxo is offline   Reply With Quote
Old 10-14-2022, 08:53 PM   #5
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Quote:
Originally Posted by mtxo View Post
I test the new version. The new code for save the markdown file to the kobo storage is ok. But, maybe it have a "bug" in the case of a non english environment. Mine is spanish, and the "Exported Annotations" dir is called "Anotaciones exportadas" in spanish. So maybe you need to put a note in the script to the user to adjust it to her language...
Oh I didn't know that, I always assumed it would be the same name. But I'll add a note.

Quote:
Originally Posted by mtxo View Post
And the result markdown file is not "the same" in the two versions. In the previous version I have a more complete file, whit probably all the highlights of the kobo saved. In the new one, there are not highlights for some books. You can see them in the attached files (i renamed them to the .txt extension). The 1.txt file is made with the first version, and the 2.txt file with the second.
Do you use epub or kepub books?

Also, could you try to remove the line
Code:
AND b.Hidden = 'false'
?

It seems like the dogears work fine, but only the highlights section is not complete.
qkqw is offline   Reply With Quote
Advert
Old 10-15-2022, 04:22 AM   #6
mtxo
Junior Member
mtxo began at the beginning.
 
Posts: 6
Karma: 22
Join Date: Oct 2022
Device: Kobo Nia
I normally use epub format, but in some cases I convert it to kepub, mostly for see the footnotes. But all the books that have highlights now are epubs (in the kobo storage I have some kepubs, but they don't have highlights).

I test the proposed change. I first remove that line in the SQL code for the highlights, and make a probe. The result is the same, no changes. Later I remove too the same line in the SQL for the dogears, and nothing changes, the resulted file is the same that in the first probe, and is the same that in the probes for yesterday.
mtxo is offline   Reply With Quote
Old 10-15-2022, 04:27 AM   #7
mtxo
Junior Member
mtxo began at the beginning.
 
Posts: 6
Karma: 22
Join Date: Oct 2022
Device: Kobo Nia
Only for completeness. The book that have the most highlights, "Informe sobre Biodiversidad", was a long M$ Word document, that I convert it to epub with Libreoffice. Later sideloaded to the kobo for carefully read it, make annotations, and export them with the Kobo's own mechanism.
mtxo is offline   Reply With Quote
Old 10-15-2022, 11:58 AM   #8
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Quote:
Originally Posted by mtxo View Post
I normally use epub format, but in some cases I convert it to kepub, mostly for see the footnotes. But all the books that have highlights now are epubs (in the kobo storage I have some kepubs, but they don't have highlights).
Thanks for pointing this out. I almost exclusively use kepub, so I didn't notice that epub uses a slightly different format for storing its chapters in the database. I've (hopefully) fixed this with a new version.

Want to try and see if it works for you?
qkqw is offline   Reply With Quote
Old 10-16-2022, 02:18 PM   #9
mtxo
Junior Member
mtxo began at the beginning.
 
Posts: 6
Karma: 22
Join Date: Oct 2022
Device: Kobo Nia
The new code works sooo well!!!

I get a file like the made with the first version, with all the highlights, and all the dogears than in the first version, with the same order too. And now (I don't know why) I have a "new" dogear that was in a PDF file, that did not appear in the first version!!

And I get too the new improvements: the chapters, the date and hour of the highlights, the icons for the highlights and dogears. I think that I have all of them!!

Attached you can see a screenshot of part of the new file, that I convert to html with pandoc, in Debian linux. You can see in it the new dogear from the pdf file.

Many thanks for all the work. It may be useful. Greetings. Mtxo.
Attached Thumbnails
Click image for larger version

Name:	snap0000.jpg
Views:	371
Size:	70.6 KB
ID:	197210  
mtxo is offline   Reply With Quote
Old 10-16-2022, 05:48 PM   #10
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Quote:
Originally Posted by mtxo View Post
The new code works sooo well!!!
You are welcome

I've just pushed a few minor changes, like showing the chapter name in current reads and adding a few newlines in order to have pandoc render the Markdown correctly.
qkqw is offline   Reply With Quote
Old 10-17-2022, 12:07 AM   #11
vinnyt
Junior Member
vinnyt began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Oct 2022
Device: kobo libra 2
Thanks for creating this. This is great!

I tested it out as well and modified it a bit to upload to my local NAS that hosts a webdav server. Everything is working however the order by statement might need a fix.

I'm thinking that it should be sorted in the following order: Book -> chapter -> pageNum instead of dateCreated in the final sort of the query as it is now. Not sure if you have access to the page number but to me it makes more sense to have the annotations in order by page number within a chapter.

Last edited by vinnyt; 10-17-2022 at 12:12 AM.
vinnyt is offline   Reply With Quote
Old 10-17-2022, 01:35 AM   #12
vinnyt
Junior Member
vinnyt began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Oct 2022
Device: kobo libra 2
I may have figured out the issue mentioned above. Chapter progress seems to do the trick.

ORDER BY c.BookTitle ASC,
c.VolumeIndex ASC,
b.ChapterProgress ASC;"
vinnyt is offline   Reply With Quote
Old 10-17-2022, 08:29 AM   #13
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Quote:
Originally Posted by vinnyt View Post
Thanks for creating this. This is great!
Thanks

Quote:
Originally Posted by vinnyt View Post
I tested it out as well and modified it a bit to upload to my local NAS that hosts a webdav server.
Happy to integrate any changes or make it easier to support other webdav/upload solutions.

Quote:
Originally Posted by vinnyt View Post
I'm thinking that it should be sorted in the following order: Book -> chapter -> pageNum instead of dateCreated in the final sort of the query as it is now. Not sure if you have access to the page number but to me it makes more sense to have the annotations in order by page number within a chapter.
I thought it make sense to order by the creation of a bookmark instead of the chapter progress, as that's my reading style. But I don't mind changing this, see the new version.

I'd love to have access to the page number. But it seems that page numbers will be calculated at runtime, depending on your current font and margin settings.
qkqw is offline   Reply With Quote
Old 10-17-2022, 04:05 PM   #14
vinnyt
Junior Member
vinnyt began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Oct 2022
Device: kobo libra 2
Looking much better. However I'm still seeing an issue with dogears bubbling to the top of the output even though it belongs to a later chapter/subchapter. Possibly a bug in the query for chapter identification for dogears.

Also here is an updated config script. I added a 'sleep 5' just before the http call in the notes.sh script. This is to make sure there is enough time for wifi to connect. The last curl command returns an error and the main NM config waits upto to 10 seconds for everything to complete.

Quote:
menu_item :main :Sync Annotations :nickel_wifi :autoconnect
chain_success :cmd_output :9999:quiet:/mnt/onboard/.adds/notes/notes.sh
chain_success:skip:2
chain_failure :dbg_toast :Failed to sync notes!
chain_always:skip:-1
chain_success :dbg_toast :Notes Sync Complete!
vinnyt is offline   Reply With Quote
Old 10-18-2022, 05:39 AM   #15
qkqw
Connoisseur
qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.qkqw is at one with the great books of the world.
 
Posts: 62
Karma: 143000
Join Date: Apr 2022
Device: Kobo Libra 2
Quote:
Originally Posted by vinnyt View Post
Looking much better. However I'm still seeing an issue with dogears bubbling to the top of the output even though it belongs to a later chapter/subchapter. Possibly a bug in the query for chapter identification for dogears.
It looks fine with my books. Does it happen for all your books or just a single one?

Quote:
Originally Posted by vinnyt View Post
Also here is an updated config script. I added a 'sleep 5' just before the http call in the notes.sh script. This is to make sure there is enough time for wifi to connect. The last curl command returns an error and the main NM config waits upto to 10 seconds for everything to complete.
Ideally we'd have to use a ping test or similar to find out if an connection is working (e.g. https://github.com/fsantini/KoboClou...get.sh#L21-L37).

The sleep however doesn't have to be included in the script, as NickelMenu supports that too:

Code:
menu_item :main :Sync Annotations :nickel_wifi :autoconnect
chain_success :cmd_output :9999:quiet:sleep 5
chain_success :cmd_output :9999:quiet:/mnt/onboard/.adds/notes/notes.sh
chain_success:skip:2
chain_failure :dbg_toast :Failed to sync notes!
chain_always:skip:-1
chain_success :dbg_toast :Notes Sync Complete!
You might also want to hook this script into KoboCloud or run it manually, hence a sleep 5 inside the script is IMHO not the best approach.
qkqw is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
import of annotations and reading progress Percivale Kobo Developer's Corner 3 12-20-2017 08:21 PM
Reading app with freehand annotations and cloud sync (Dropbox, Google Drive, ...) sjvs Reading and Management 1 08-30-2013 03:18 PM
Any way of exporting bookmarks in Shubooks? robinson Apple Devices 4 03-31-2013 11:23 AM
Importing/exporting bookmarks riki Library Management 0 01-18-2012 10:25 AM
Exporting Annotations musemj6 Kindle Developer's Corner 0 12-03-2008 01:01 PM


All times are GMT -4. The time now is 01:18 PM.


MobileRead.com is a privately owned, operated and funded community.