Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Amazon Kindle > Kindle Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 05-14-2010, 11:29 AM   #1
ngukho
Junior Member
ngukho began at the beginning.
 
ngukho's Avatar
 
Posts: 1
Karma: 34
Join Date: Sep 2009
Device: Kindle DX
Collections data format in 2.5 firmware

Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.

The "collections.json" is a standard json file. The only tricky part is how it identifies a document --- or what do those hashes come from. A little trial and error shows that they are merely the sha1 hash value of the document's absolute path, say a H.pdf is at the Kindle's documents directory, then its hash in collections.json would be sha1 hash of the string "/mnt/us/documents/H.pdf".

Voila, all that's needed for a PC-end collections manager is above. Anyone willing to contribute a GUI for this job? (-8

Last edited by ngukho; 05-14-2010 at 11:38 AM.
ngukho is offline   Reply With Quote
Old 05-15-2010, 05:56 PM   #2
elmoglick
Groupie
elmoglick doesn't litterelmoglick doesn't litterelmoglick doesn't litter
 
Posts: 165
Karma: 206
Join Date: Dec 2007
Location: Kansas City
Device: Kindle1, Kindle DX, Kindle DXG
Quote:
Originally Posted by ngukho View Post
Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.
What is interesting is that deleting the file results in its immediate re-creation. From where??
elmoglick is offline   Reply With Quote
Advert
Old 05-20-2010, 11:54 AM   #3
=X=
Wizard
=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.=X= ought to be getting tired of karma fortunes by now.
 
=X='s Avatar
 
Posts: 3,671
Karma: 12205348
Join Date: Mar 2008
Device: Galaxy S, Nook w/CM7
Couldn't you use calibre for the GUI?
=X= is offline   Reply With Quote
Old 05-20-2010, 05:58 PM   #4
Pushka
Wizard
Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.Pushka ought to be getting tired of karma fortunes by now.
 
Pushka's Avatar
 
Posts: 1,119
Karma: 1019140
Join Date: Oct 2009
Location: Australia
Device: kindle, Ipad, Iphone, Nexus and PPW
That is like the screensavers. You delete them from the kindle, and next time, they are back! Calibre doesnt set up the collections - the developers dont even have the upgrade yet.
Pushka is offline   Reply With Quote
Old 06-18-2010, 01:56 PM   #5
schaefer
Junior Member
schaefer began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2010
Device: Kindle DX
Quote:
Originally Posted by ngukho View Post
Managing collections on Kindle is painful and slow. However, if you dig into the new file system, you could find a "collections.json" at /mnt/us/system, which stores all the stuff related to collections data.

The "collections.json" is a standard json file. The only tricky part is how it identifies a document --- or what do those hashes come from. A little trial and error shows that they are merely the sha1 hash value of the document's absolute path, say a H.pdf is at the Kindle's documents directory, then its hash in collections.json would be sha1 hash of the string "/mnt/us/documents/H.pdf".

Voila, all that's needed for a PC-end collections manager is above. Anyone willing to contribute a GUI for this job? (-8
I have not been able to replicate this result. I am using OS X and the following command to generate the hash.

echo <Filepath> | openssl sha1

The result I get is the same length as the hash in the collections file but is not equal. Additionally, each item in my collections file has an asterisk before the hash, are others seeing this as well?
schaefer is offline   Reply With Quote
Advert
Old 06-18-2010, 02:23 PM   #6
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 13,477
Karma: 26012492
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
Use echo -n

(no trailing linebreak) [Provided OS X's echo implementation is more or less the same as on Linux]

And yep, I also have an asterisk here.

Last edited by NiLuJe; 06-18-2010 at 02:25 PM.
NiLuJe is offline   Reply With Quote
Old 06-18-2010, 02:41 PM   #7
schaefer
Junior Member
schaefer began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2010
Device: Kindle DX
Quote:
Originally Posted by NiLuJe View Post
Use echo -n

(no trailing linebreak) [Provided OS X's echo implementation is more or less the same as on Linux]

And yep, I also have an asterisk here.
Still a different result. Here's the command I'm using

echo -n "/mnt/us/documents/hackermonthly-issue1.pdf" | openssl sha1
schaefer is offline   Reply With Quote
Old 06-18-2010, 02:56 PM   #8
clarknova
Addict
clarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with others
 
clarknova's Avatar
 
Posts: 241
Karma: 2617
Join Date: Mar 2009
Location: Greenwood, SC
Device: Kindle 2
Quote:
Originally Posted by schaefer View Post
Still a different result. Here's the command I'm using

echo -n "/mnt/us/documents/hackermonthly-issue1.pdf" | openssl sha1
That similar command line works for my collections on linux, and the above gives "1d71ee1d02b3a543980d690d8d4026a8be73e3ca" as does using sha1sum instead of openssl.

According to the Mac OS X man pages, echo is posix compatible and supports the -n flag as it should.
clarknova is offline   Reply With Quote
Old 06-18-2010, 02:58 PM   #9
schaefer
Junior Member
schaefer began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2010
Device: Kindle DX
Unhappy

Quote:
Originally Posted by clarknova View Post
That similar command line works for my collections on linux, and the above gives "1d71ee1d02b3a543980d690d8d4026a8be73e3ca" as does using sha1sum instead of openssl.

According to the Mac OS X man pages, echo is posix compatible and supports the -n flag as it should.
That's exactly the result I get as well, however it isn't the one that is in the collections.json file, hence my frustration.

EDIT:

Would other modifications have effected the way in which my DX creates the hashes? I have installed the updated screensaver hack.

Last edited by schaefer; 06-18-2010 at 03:11 PM.
schaefer is offline   Reply With Quote
Old 06-18-2010, 03:51 PM   #10
schaefer
Junior Member
schaefer began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2010
Device: Kindle DX
Quote:
Originally Posted by schaefer View Post
That's exactly the result I get as well, however it isn't the one that is in the collections.json file, hence my frustration.

EDIT:

Would other modifications have effected the way in which my DX creates the hashes? I have installed the updated screensaver hack.
Update: I just tried it with a few other files and I get the same hash as the Kindle creates. The difference I notice is that these other files don't have hyphens in the filename. Is it possible that the hyphen is causing the difference?
schaefer is offline   Reply With Quote
Old 06-18-2010, 03:54 PM   #11
clarknova
Addict
clarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with others
 
clarknova's Avatar
 
Posts: 241
Karma: 2617
Join Date: Mar 2009
Location: Greenwood, SC
Device: Kindle 2
Quote:
Originally Posted by schaefer View Post
Update: I just tried it with a few other files and I get the same hash as the Kindle creates. The difference I notice is that these other files don't have hyphens in the filename. Is it possible that the hyphen is causing the difference?
Could be. It might be encoded. What's the hash supposed to be?

However, looking at mine, I have two files with hyphens, and they encode fine?
clarknova is offline   Reply With Quote
Old 06-18-2010, 03:55 PM   #12
schaefer
Junior Member
schaefer began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2010
Device: Kindle DX
Quote:
Originally Posted by clarknova View Post
Could be. It might be encoded. What's the hash supposed to be?
08636757653b8fe7a4f0fff42c4160c809aeba87
schaefer is offline   Reply With Quote
Old 06-18-2010, 04:05 PM   #13
clarknova
Addict
clarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with othersclarknova plays well with others
 
clarknova's Avatar
 
Posts: 241
Karma: 2617
Join Date: Mar 2009
Location: Greenwood, SC
Device: Kindle 2
Quote:
Originally Posted by schaefer View Post
08636757653b8fe7a4f0fff42c4160c809aeba87
Weird. I went and downloaded that pdf, and put it on my kindle (named exactly hackermonthly-issue1.pdf) and added to a collection and I see:
"*1d71ee1d02b3a543980d690d8d4026a8be73e3ca"

Just like the sha1 hash I get from the command line.

BTW - Thanks for pointing out the magazine, looks interesting.
clarknova is offline   Reply With Quote
Old 06-18-2010, 04:12 PM   #14
schaefer
Junior Member
schaefer began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2010
Device: Kindle DX
Quote:
Originally Posted by clarknova View Post
Weird. I went and downloaded that pdf, and put it on my kindle (named exactly hackermonthly-issue1.pdf) and added to a collection and I see:
"*1d71ee1d02b3a543980d690d8d4026a8be73e3ca"

Just like the sha1 hash I get from the command line.

BTW - Thanks for pointing out the magazine, looks interesting.
Do you think it could be a Kindle 2 vs. DX difference? I may delete the collections and files and copy the files and try again when I can get back to my Kindle.

Last edited by schaefer; 06-18-2010 at 04:14 PM.
schaefer is offline   Reply With Quote
Old 09-30-2010, 08:54 PM   #15
Daniel Mahler
Junior Member
Daniel Mahler began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Sep 2010
Device: Kindle DX Graphite
Quote:
Originally Posted by schaefer View Post
Do you think it could be a Kindle 2 vs. DX difference? I may delete the collections and files and copy the files and try again when I can get back to my Kindle.
Has anybody figured out what kind of escaping is applied to non-alphanumeric chars before applying sha1.
Daniel Mahler is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
DR800 Any official information about firmware 2.0 release data? overdrunk iRex 36 06-03-2010 02:35 PM
Add books - Meta Data format wwaldo Calibre 2 03-17-2010 07:38 PM
Large eBook collections post PRS-500 EPUB firmware upgrade AprilHare Sony Reader 0 12-18-2009 05:47 PM
Migrating Calibre from a failing hard drive - mangled my Collections data lconover Calibre 4 03-13-2009 08:25 AM
.PDB Encrytpion tool/"F-Format Wrapper Package"/Published Data arvinder Workshop 0 08-20-2007 04:17 AM


All times are GMT -4. The time now is 04:49 PM.


MobileRead.com is a privately owned, operated and funded community.