LibrarianSync: Kindle collections builder

[bigwoof]

*UPDATE*

can't figure this out yet. changing the update from

"members": [0, 1, 2]

to 

"SET members": "members || [0, 1 ,2]"

is not being accepted by the post command. will keep trying. reading /usr/lib/ccat/change.lua to see if there is some other way as well

does anyone have a better way to do this? another solution is to upgrade the sqlite3 on the kindle from 3.26.0 to 3.32 or above (that has a SQL limit of 32K instead of 999) or recompile the existing sqlite3 with a highest SQLITE_MAX_VARIABLE_NUMBER but changing the sqlite binary seems like a pretty drastic change..


*UPDATE Dec 30th 2023*

I had some time this morning so I dug deeper to figure out the issue. after adding debugging to both cc_update.py (in librariansync) and in change.lua (in /usr/lib/ccat), it appears that the issue is that change.lua assumes that *every* update is an overwrite. it basically deletes any books in the collection that is not in the current update message.

this coupled with the sqlite 999 variable length limit means that the code, as is, as far as I can tell, will never work for larger collections.

I got it to work by commenting out the delete code in both change_collections and change_members in change.lua but that's not a scalable change..

it seems like a more scalable solution is to rewrite change.lua to add some flag to not delete (or some other way to handle a large number of related updates to the same collection) or to update sqlite to allow for large variable lengths. I'm not sure how to do either, to be honest. so I'm just going to leave it here..

[bigwoof]

so to summarise, I think I found two different bugs.

1. SQLITE_MAX_VARIABLE_NUMBER is set to 999 or lower (did not verify the exact number)

symptoms: if your collection has more books than this number, it will crash during the update with a "too many variables" error

solution: fix sqlite to have a higher limit (not sure how to do this myself)

my hacked solution (see previous post for details):
i) patch generate_collections.py to generate update sets that are 400 entries or lower
ii) comment out all the delete commands in change.lua

2. if you have a lot of books and collections (I am updating ~160 collections with about 4,000 books), the kindle will reboot during the update (out of memory I suspect)

Solution: patch cc_update.py to send only 100 SQL commands at a time to change.lua. this works for me and my collections updated

my fix is below (change the code at the bottom of cc_update.py)

            for i in range(0,len(self.commands),100):
                new_commands = self.commands[i:i+100]
                full_command = {"commands": new_commands,
                                "type": "ChangeRequest", "id": 1}
              
                r = requests.post("http://127.0.0.1:9101/change",
                                  data=json.dumps(full_command),
                                  headers={'AuthToken': self.session_token, 'con
tent-type':  'application/json'},
                                   proxies={'no': 'pass'})
                if r.status_code == requests.codes.ok:
                     log(LIBRARIAN_SYNC, "cc_update", "Success.")
                else:
                     log(LIBRARIAN_SYNC, "cc_update", "Oh, no. It failed.", "E")

[Zix]

I have read before that this can work on a PW5 2023 and I had been waiting for this for months to jailbreak the device but despite everything I am not able to make it work.


1- I have a PW5 with jailbreak applied
2- I have installed Python on the Kindle
3- I have installed Kindle Collections in Calibre, I have applied the Qt patch and I have all my series well organized
4- I have installed LybrarianSync throught MRI and I have verified that the collections.json that the plugin creates for me is correctly installed in system/collections.json.

However, when I click rebuild collections (from calibre) in LybrarianSync (booted from KUAL) it stays stuck and although I have read that it still generated it that way in my case, no matter how much I wait, I don't have any collection.

I don't know if anyone has found the solution and can share it. Thank you

[bigwoof]

Originally Posted by Zix

I don't know if anyone has found the solution and can share it. Thank you

one way to test if the sync is working is to just have 1 collection with about 10 books. see if that works. if it does, then any further errors will be from two different sources.

1. you did not wait long enough

Note: updating a lot of books and/or collections takes a long time btw. an hour or two for my case with about 4000 books and 150 collections.

nothing will display in kual (it will look like librariansync has just finished or died) but it is running in the background and you will notice this via the log messages or checking that the generate_collections.py process is running..

you can get the log messages by ssh ing into the kindle and running

showlog -f | grep -i LIBRARIAN

you can see the process by running (at the shell command line after sshing in)

ps aw | grep -i generate

2. you have too many books

if your collections have more than 900 or so books in them, then it will fail with a SQL error printed in the log. the solution (as stated in my last post) is to send smaller SQL sets and patch change.lua to not delete existing libraries. I can send you my code if you want but I can't guarantee it won't break anything else. it works for me though

3. if the simple sync did not work, what does the log print (showlog -f | grep -i LIBRARIAN)?

[Zix]

Thank you!

It seems like it was just a waiting problem. In my case I had waited 30 minutes and I only have 30 collections and it had not worked for me.

After reading your comment I have left it for 2 hours and although I have never gotten a different message, I have clicked on / and everything has been correct.

Very thankful


P.D: Sorry for Offtopic...There is a way to put image covers in Collections view? Damn..looks so ugly now

[PaigheTurn]

Does this work for Kindle 3? I have a blacklisted K3W from a junkshop. So I cant register for collections.

[Ruskie_it]

Originally Posted by bigwoof

one way to test if the sync is working is to just have 1 collection with about 10 books. see if that works. if it does, then any further errors will be from two different sources.

1. you did not wait long enough

Note: updating a lot of books and/or collections takes a long time btw. an hour or two for my case with about 4000 books and 150 collections.

nothing will display in kual (it will look like librariansync has just finished or died) but it is running in the background and you will notice this via the log messages or checking that the generate_collections.py process is running..

you can get the log messages by ssh ing into the kindle and running

showlog -f | grep -i LIBRARIAN

you can see the process by running (at the shell command line after sshing in)

ps aw | grep -i generate

2. you have too many books

if your collections have more than 900 or so books in them, then it will fail with a SQL error printed in the log. the solution (as stated in my last post) is to send smaller SQL sets and patch change.lua to not delete existing libraries. I can send you my code if you want but I can't guarantee it won't break anything else. it works for me though

3. if the simple sync did not work, what does the log print (showlog -f | grep -i LIBRARIAN)?

You got it, according to OP answer... yet I was extremely puzzled: it never takes more than 3-4 seconds in my PW5... Sure I don't have 4k books, but I have almost a thousand...

[bigwoof]

Originally Posted by Ruskie_it

You got it, according to OP answer... yet I was extremely puzzled: it never takes more than 3-4 seconds in my PW5... Sure I don't have 4k books, but I have almost a thousand...

what firmware are you on? on 5.16.2.1.1 (what I'm using on my paperwhite 32G PW5), the reading of the json collection file takes about 20-30 minutes and about 60-90 minutes for the SQL

the sql commands are sent as a POST message and then the code just waits for a long time before the post request returns. on the other side change.lua reads the post and issues a valid sql commit request. which takes a long time... not sure how to make this faster.

The kindle just seems quite slow in general.. but yes this is really slow..

[hondabf]

Originally Posted by PaigheTurn

Does this work for Kindle 3? I have a blacklisted K3W from a junkshop. So I cant register for collections.

I got it running on my jailbreaked PW4 with 5.10.0.2 firmware, I also haven't connected to the Internet :-)