MobileRead Forums - View Single Post - LibrarianSync: Kindle collections builder

bigwoof · 12-23-2023, 09:12 PM

*UPDATE*

can't figure this out yet. changing the update from

Code:

"members": [0, 1, 2]

to 

"SET members": "members || [0, 1 ,2]"

is not being accepted by the post command. will keep trying. reading /usr/lib/ccat/change.lua to see if there is some other way as well

does anyone have a better way to do this? another solution is to upgrade the sqlite3 on the kindle from 3.26.0 to 3.32 or above (that has a SQL limit of 32K instead of 999) or recompile the existing sqlite3 with a highest SQLITE_MAX_VARIABLE_NUMBER but changing the sqlite binary seems like a pretty drastic change..

*UPDATE Dec 30th 2023*

I had some time this morning so I dug deeper to figure out the issue. after adding debugging to both cc_update.py (in librariansync) and in change.lua (in /usr/lib/ccat), it appears that the issue is that change.lua assumes that *every* update is an overwrite. it basically deletes any books in the collection that is not in the current update message.

this coupled with the sqlite 999 variable length limit means that the code, as is, as far as I can tell, will never work for larger collections.

I got it to work by commenting out the delete code in both change_collections and change_members in change.lua but that's not a scalable change..

it seems like a more scalable solution is to rewrite change.lua to add some flag to not delete (or some other way to handle a large number of related updates to the same collection) or to update sqlite to allow for large variable lengths. I'm not sure how to do either, to be honest. so I'm just going to leave it here..

12-23-2023, 09:12 PM	#571
bigwoof Connoisseur Posts: 88 Karma: 18 Join Date: Jun 2020 Device: Boox Note Air3, don't use my kindle anymore	UPDATE can't figure this out yet. changing the update from Code: "members": [0, 1, 2] to "SET members": "members \|\| [0, 1 ,2]" is not being accepted by the post command. will keep trying. reading /usr/lib/ccat/change.lua to see if there is some other way as well does anyone have a better way to do this? another solution is to upgrade the sqlite3 on the kindle from 3.26.0 to 3.32 or above (that has a SQL limit of 32K instead of 999) or recompile the existing sqlite3 with a highest SQLITE_MAX_VARIABLE_NUMBER but changing the sqlite binary seems like a pretty drastic change.. UPDATE Dec 30th 2023 I had some time this morning so I dug deeper to figure out the issue. after adding debugging to both cc_update.py (in librariansync) and in change.lua (in /usr/lib/ccat), it appears that the issue is that change.lua assumes that every update is an overwrite. it basically deletes any books in the collection that is not in the current update message. this coupled with the sqlite 999 variable length limit means that the code, as is, as far as I can tell, will never work for larger collections. I got it to work by commenting out the delete code in both change_collections and change_members in change.lua but that's not a scalable change.. it seems like a more scalable solution is to rewrite change.lua to add some flag to not delete (or some other way to handle a large number of related updates to the same collection) or to update sqlite to allow for large variable lengths. I'm not sure how to do either, to be honest. so I'm just going to leave it here.. Last edited by bigwoof; 12-29-2023 at 10:45 PM.