View Full Version : Collections data format in 2.5 firmware


ngukho
05-14-2010, 11:29 AM
Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.

The "collections.json" is a standard json file. The only tricky part is how it identifies a document --- or what do those hashes come from. A little trial and error shows that they are merely the sha1 hash value of the document's absolute path, say a H.pdf is at the Kindle's documents directory, then its hash in collections.json would be sha1 hash of the string "/mnt/us/documents/H.pdf".

Voila, all that's needed for a PC-end collections manager is above. Anyone willing to contribute a GUI for this job? (-8

elmoglick
05-15-2010, 05:56 PM
Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.


What is interesting is that deleting the file results in its immediate re-creation. From where??

=X=
05-20-2010, 11:54 AM
Couldn't you use calibre for the GUI?

Pushka
05-20-2010, 05:58 PM
That is like the screensavers. You delete them from the kindle, and next time, they are back! Calibre doesnt set up the collections - the developers dont even have the upgrade yet.

schaefer
06-18-2010, 01:56 PM
Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.

The "collections.json" is a standard json file. The only tricky part is how it identifies a document --- or what do those hashes come from. A little trial and error shows that they are merely the sha1 hash value of the document's absolute path, say a H.pdf is at the Kindle's documents directory, then its hash in collections.json would be sha1 hash of the string "/mnt/us/documents/H.pdf".

Voila, all that's needed for a PC-end collections manager is above. Anyone willing to contribute a GUI for this job? (-8

I have not been able to replicate this result. I am using OS X and the following command to generate the hash.

echo <Filepath> | openssl sha1

The result I get is the same length as the hash in the collections file but is not equal. Additionally, each item in my collections file has an asterisk before the hash, are others seeing this as well?

NiLuJe
06-18-2010, 02:23 PM
Use echo -n

(no trailing linebreak) [Provided OS X's echo implementation is more or less the same as on Linux]

And yep, I also have an asterisk here.

schaefer
06-18-2010, 02:41 PM
Use echo -n

(no trailing linebreak) [Provided OS X's echo implementation is more or less the same as on Linux]

And yep, I also have an asterisk here.

Still a different result. Here's the command I'm using

echo -n "/mnt/us/documents/hackermonthly-issue1.pdf" | openssl sha1

clarknova
06-18-2010, 02:56 PM
Still a different result. Here's the command I'm using

echo -n "/mnt/us/documents/hackermonthly-issue1.pdf" | openssl sha1

That similar command line works for my collections on linux, and the above gives "1d71ee1d02b3a543980d690d8d4026a8be73e3ca" as does using sha1sum instead of openssl.

According to the Mac OS X man pages, echo is posix compatible and supports the -n flag as it should.

schaefer
06-18-2010, 02:58 PM
That similar command line works for my collections on linux, and the above gives "1d71ee1d02b3a543980d690d8d4026a8be73e3ca" as does using sha1sum instead of openssl.

According to the Mac OS X man pages, echo is posix compatible and supports the -n flag as it should.

That's exactly the result I get as well, however it isn't the one that is in the collections.json file, hence my frustration.

EDIT:

Would other modifications have effected the way in which my DX creates the hashes? I have installed the updated screensaver hack.

schaefer
06-18-2010, 03:51 PM
That's exactly the result I get as well, however it isn't the one that is in the collections.json file, hence my frustration.

EDIT:

Would other modifications have effected the way in which my DX creates the hashes? I have installed the updated screensaver hack.

Update: I just tried it with a few other files and I get the same hash as the Kindle creates. The difference I notice is that these other files don't have hyphens in the filename. Is it possible that the hyphen is causing the difference?

clarknova
06-18-2010, 03:54 PM
Update: I just tried it with a few other files and I get the same hash as the Kindle creates. The difference I notice is that these other files don't have hyphens in the filename. Is it possible that the hyphen is causing the difference?

Could be. It might be encoded. What's the hash supposed to be?

However, looking at mine, I have two files with hyphens, and they encode fine?

schaefer
06-18-2010, 03:55 PM
Could be. It might be encoded. What's the hash supposed to be?

08636757653b8fe7a4f0fff42c4160c809aeba87

clarknova
06-18-2010, 04:05 PM
08636757653b8fe7a4f0fff42c4160c809aeba87

Weird. I went and downloaded that pdf, and put it on my kindle (named exactly hackermonthly-issue1.pdf) and added to a collection and I see:
"*1d71ee1d02b3a543980d690d8d4026a8be73e3ca"

Just like the sha1 hash I get from the command line.

BTW - Thanks for pointing out the magazine, looks interesting.

schaefer
06-18-2010, 04:12 PM
Weird. I went and downloaded that pdf, and put it on my kindle (named exactly hackermonthly-issue1.pdf) and added to a collection and I see:
"*1d71ee1d02b3a543980d690d8d4026a8be73e3ca"

Just like the sha1 hash I get from the command line.

BTW - Thanks for pointing out the magazine, looks interesting.

Do you think it could be a Kindle 2 vs. DX difference? I may delete the collections and files and copy the files and try again when I can get back to my Kindle.

Daniel Mahler
09-30-2010, 08:54 PM
Do you think it could be a Kindle 2 vs. DX difference? I may delete the collections and files and copy the files and try again when I can get back to my Kindle.

Has anybody figured out what kind of escaping is applied to non-alphanumeric chars before applying sha1.

NiLuJe
09-30-2010, 09:58 PM
@Daniel Mahler: No idea. UTF-8? htmlentities?

mickeymicks
10-27-2010, 11:48 AM
Hi, i want to thank this thread for the info about the json file format. i'm trying to make a GUI for this and been using it on my kindle. i've posted it on my blog if anyone is interested (mickeymickstechblog.blogspot.com).
umm, but it is for Windows only. and i haven't tested/tried it on any other kindle.
note: sorry for plugging in my blog. the admin can delete it, but pls do leave the "thank u" part. hehe

happy reading!

ElizabethN
10-27-2010, 01:24 PM
Hi, i want to thank this thread for the info about the json file format. i'm trying to make a GUI for this and been using it on my kindle. i've posted it on my blog if anyone is interested (mickeymickstechblog.blogspot.com).
umm, but it is for Windows only. and i haven't tested/tried it on any other kindle.
note: sorry for plugging in my blog. the admin can delete it, but pls do leave the "thank u" part. hehe

happy reading!

Have you looked into this thread at mobileread? Firmware Update Kindle Collection Manager for Windows, url hopefully below:

http://www.mobileread.com/forums/showthread.php?t=94261

The developer is up to collections editor v.4 and working on v.5. As someone with over 2500 files loaded on the kindle, I've found his program very useful. It's still not without flaws but he is working on them and improvements with each release.

I downloaded your editor but probably won't have a chance to try it until a few days from now, depending on work schedule.

Thanks for sharing.

mickeymicks
10-29-2010, 11:19 PM
thanks ElizabethN for sharing the link!
i'll check out the editor as soon as i can :)