View Full Version : Collections data format in 2.5 firmware
ngukho 05-14-2010, 11:29 AM Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.
The "collections.json" is a standard json file. The only tricky part is how it identifies a document --- or what do those hashes come from. A little trial and error shows that they are merely the sha1 hash value of the document's absolute path, say a H.pdf is at the Kindle's documents directory, then its hash in collections.json would be sha1 hash of the string "/mnt/us/documents/H.pdf".
Voila, all that's needed for a PC-end collections manager is above. Anyone willing to contribute a GUI for this job? (-8
elmoglick 05-15-2010, 05:56 PM Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.
What is interesting is that deleting the file results in its immediate re-creation. From where??
Couldn't you use calibre for the GUI?
Pushka 05-20-2010, 05:58 PM That is like the screensavers. You delete them from the kindle, and next time, they are back! Calibre doesnt set up the collections - the developers dont even have the upgrade yet.
schaefer 06-18-2010, 01:56 PM Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.
The "collections.json" is a standard json file. The only tricky part is how it identifies a document --- or what do those hashes come from. A little trial and error shows that they are merely the sha1 hash value of the document's absolute path, say a H.pdf is at the Kindle's documents directory, then its hash in collections.json would be sha1 hash of the string "/mnt/us/documents/H.pdf".
Voila, all that's needed for a PC-end collections manager is above. Anyone willing to contribute a GUI for this job? (-8
I have not been able to replicate this result. I am using OS X and the following command to generate the hash.
echo <Filepath> | openssl sha1
The result I get is the same length as the hash in the collections file but is not equal. Additionally, each item in my collections file has an asterisk before the hash, are others seeing this as well?
NiLuJe 06-18-2010, 02:23 PM Use echo -n
(no trailing linebreak) [Provided OS X's echo implementation is more or less the same as on Linux]
And yep, I also have an asterisk here.
schaefer 06-18-2010, 02:41 PM Use echo -n
(no trailing linebreak) [Provided OS X's echo implementation is more or less the same as on Linux]
And yep, I also have an asterisk here.
Still a different result. Here's the command I'm using
echo -n "/mnt/us/documents/hackermonthly-issue1.pdf" | openssl sha1
clarknova 06-18-2010, 02:56 PM Still a different result. Here's the command I'm using
echo -n "/mnt/us/documents/hackermonthly-issue1.pdf" | openssl sha1
That similar command line works for my collections on linux, and the above gives "1d71ee1d02b3a543980d690d8d4026a8be73e3ca" as does using sha1sum instead of openssl.
According to the Mac OS X man pages, echo is posix compatible and supports the -n flag as it should.
schaefer 06-18-2010, 02:58 PM That similar command line works for my collections on linux, and the above gives "1d71ee1d02b3a543980d690d8d4026a8be73e3ca" as does using sha1sum instead of openssl.
According to the Mac OS X man pages, echo is posix compatible and supports the -n flag as it should.
That's exactly the result I get as well, however it isn't the one that is in the collections.json file, hence my frustration.
EDIT:
Would other modifications have effected the way in which my DX creates the hashes? I have installed the updated screensaver hack.
schaefer 06-18-2010, 03:51 PM That's exactly the result I get as well, however it isn't the one that is in the collections.json file, hence my frustration.
EDIT:
Would other modifications have effected the way in which my DX creates the hashes? I have installed the updated screensaver hack.
Update: I just tried it with a few other files and I get the same hash as the Kindle creates. The difference I notice is that these other files don't have hyphens in the filename. Is it possible that the hyphen is causing the difference?
clarknova 06-18-2010, 03:54 PM Update: I just tried it with a few other files and I get the same hash as the Kindle creates. The difference I notice is that these other files don't have hyphens in the filename. Is it possible that the hyphen is causing the difference?
Could be. It might be encoded. What's the hash supposed to be?
However, looking at mine, I have two files with hyphens, and they encode fine?
schaefer 06-18-2010, 03:55 PM Could be. It might be encoded. What's the hash supposed to be?
08636757653b8fe7a4f0fff42c4160c809aeba87
clarknova 06-18-2010, 04:05 PM 08636757653b8fe7a4f0fff42c4160c809aeba87
Weird. I went and downloaded that pdf, and put it on my kindle (named exactly hackermonthly-issue1.pdf) and added to a collection and I see:
"*1d71ee1d02b3a543980d690d8d4026a8be73e3ca"
Just like the sha1 hash I get from the command line.
BTW - Thanks for pointing out the magazine, looks interesting.
schaefer 06-18-2010, 04:12 PM Weird. I went and downloaded that pdf, and put it on my kindle (named exactly hackermonthly-issue1.pdf) and added to a collection and I see:
"*1d71ee1d02b3a543980d690d8d4026a8be73e3ca"
Just like the sha1 hash I get from the command line.
BTW - Thanks for pointing out the magazine, looks interesting.
Do you think it could be a Kindle 2 vs. DX difference? I may delete the collections and files and copy the files and try again when I can get back to my Kindle.
Daniel Mahler 09-30-2010, 08:54 PM Do you think it could be a Kindle 2 vs. DX difference? I may delete the collections and files and copy the files and try again when I can get back to my Kindle.
Has anybody figured out what kind of escaping is applied to non-alphanumeric chars before applying sha1.
NiLuJe 09-30-2010, 09:58 PM @Daniel Mahler: No idea. UTF-8? htmlentities?
mickeymicks 10-27-2010, 11:48 AM Hi, i want to thank this thread for the info about the json file format. i'm trying to make a GUI for this and been using it on my kindle. i've posted it on my blog if anyone is interested (mickeymickstechblog.blogspot.com).
umm, but it is for Windows only. and i haven't tested/tried it on any other kindle.
note: sorry for plugging in my blog. the admin can delete it, but pls do leave the "thank u" part. hehe
happy reading!
ElizabethN 10-27-2010, 01:24 PM Hi, i want to thank this thread for the info about the json file format. i'm trying to make a GUI for this and been using it on my kindle. i've posted it on my blog if anyone is interested (mickeymickstechblog.blogspot.com).
umm, but it is for Windows only. and i haven't tested/tried it on any other kindle.
note: sorry for plugging in my blog. the admin can delete it, but pls do leave the "thank u" part. hehe
happy reading!
Have you looked into this thread at mobileread? Firmware Update Kindle Collection Manager for Windows, url hopefully below:
http://www.mobileread.com/forums/showthread.php?t=94261
The developer is up to collections editor v.4 and working on v.5. As someone with over 2500 files loaded on the kindle, I've found his program very useful. It's still not without flaws but he is working on them and improvements with each release.
I downloaded your editor but probably won't have a chance to try it until a few days from now, depending on work schedule.
Thanks for sharing.
mickeymicks 10-29-2010, 11:19 PM thanks ElizabethN for sharing the link!
i'll check out the editor as soon as i can :)
|