Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Kindle Formats

Notices

Reply
 
Thread Tools Search this Thread
Old 09-08-2013, 04:37 AM   #571
pdurrant
The Grand Mouse 高貴的老鼠
pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.
 
pdurrant's Avatar
 
Posts: 71,406
Karma: 305065800
Join Date: Jul 2007
Location: Norfolk, England
Device: Kindle Voyage
Quote:
Originally Posted by Julius Caesar View Post
When I unpack a book with Kindle Unpack and repack it with Kindlegen, I got a warning: "No Page map found in the book" and search by page number is no longer available for that particular book. The page numbers are gone even though I didn't make any modification to the source files after unpacking and even though reference to page numbers are preserved in the original source files. After unpacking, I could still see a reference in each xhtml file like <a id="page88">. So, I don't know what's going on.

What is a page map and what does it look like?

How can I make my own if it's missing?

Is there anyway I can unpack and repack a Kindle book without breaking the page numbers?
Not until someone amends KindleUnpack to extract the pagemap into the format that KindleGen expects.
pdurrant is offline   Reply With Quote
Old 09-08-2013, 07:13 AM   #572
Doitsu
Grand Sorcerer
Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.
 
Doitsu's Avatar
 
Posts: 5,582
Karma: 22735033
Join Date: Dec 2010
Device: Kindle PW2
Quote:
Originally Posted by pdurrant View Post
Not until someone amends KindleUnpack to extract the pagemap into the format that KindleGen expects.
Even then it wouldn't work, since page numbers are stored in a separate .apnx file that KindleGen cannot generate, and, AFAIK, even though the APNX format has been reverse-engineered by user_none, no tool exists that'll generate a separate .apnx file with accurate page numbers from an NCX pagemap or a mobi file that was compiled with an NCX pagelist in it.
(Calibre can only generate algorithm-based .apnx files with page numbers.)

Last edited by Doitsu; 09-08-2013 at 03:28 PM.
Doitsu is offline   Reply With Quote
Advert
Old 09-08-2013, 02:44 PM   #573
Julius Caesar
Addict
Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.
 
Posts: 341
Karma: 1244808
Join Date: Oct 2012
Device: Kindle Paperwhite
Quote:
Originally Posted by Doitsu View Post
Even then it wouldn't work, since page numbers are stored in a separate .apnx file that KindleGen cannot generate, and, AFAIK, even though the APNX format has been reverse-engineered by user_none, no tool exists that'll generate a separate .apnx file with accurate page numbers from an NCX pagemap or a mobi file that was compiled with an NCX pagemap in it.
(Calibre can only generate algorithm-based .apnx files with page numbers.)
My xhtml files have references like <a id="page88">. Why can't Kindle just use that to refer to a location of a specific page number?
Julius Caesar is offline   Reply With Quote
Old 09-08-2013, 03:28 PM   #574
Doitsu
Grand Sorcerer
Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.
 
Doitsu's Avatar
 
Posts: 5,582
Karma: 22735033
Join Date: Dec 2010
Device: Kindle PW2
Quote:
Originally Posted by Julius Caesar View Post
My xhtml files have references like <a id="page88">. Why can't Kindle just use that to refer to a location of a specific page number?
If someone updated the KindleUnpack code to rebuild a pagelist section in the .ncx file and KindleGen were able to generate an .apnx file, it'd be theoretically possible to recreate the page numbers defined in the original file.
However, since very few Kindle books contain "real page numbers" and it isn't possible to generate .apnx files with KindleGen, it's very unlikely that the KindleUnpack developers will add the necessary code.
Doitsu is offline   Reply With Quote
Old 09-08-2013, 04:27 PM   #575
pdurrant
The Grand Mouse 高貴的老鼠
pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.
 
pdurrant's Avatar
 
Posts: 71,406
Karma: 305065800
Join Date: Jul 2007
Location: Norfolk, England
Device: Kindle Voyage
Quote:
Originally Posted by Julius Caesar View Post
My xhtml files have references like <a id="page88">. Why can't Kindle just use that to refer to a location of a specific page number?
Are you sure that all the page information is in your original Kindle file? Have you tried completely deleting the book from your device, along with all auxiliary files, and then putting the book back on the device (while keeping the device from connecting to the internet), to check that the page info is still found.
pdurrant is offline   Reply With Quote
Advert
Old 09-19-2013, 01:32 PM   #576
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 13,477
Karma: 26012492
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
Another small patch that follows this Calibre commit, because I managed to dig up a file like this (one of the very first Kindle book I purchased, an old Mobi v6 file), and, yep, KindleUnpack blows up on it .

(The patch from #563 is folded into this, because I'm lazy )

Code:
--- KindleUnpack_v62-ori/lib/kindleunpack.py	2013-04-22 22:04:34.000000000 +0200
+++ KindleUnpack_v62-patch/lib/kindleunpack.py	2013-09-19 19:25:29.674186532 +0200
@@ -1006,6 +1006,10 @@ class MobiHeader:
                         addValue(name, str(value))
                     elif size == 12:
                         value, = struct.unpack('>L',content)
+                        # Handle NULL ptr for *Offset...
+                        if name[-6:] == 'Offset':
+                            if value == 0xffffffff:
+                                value = 0
                         addValue(name, str(value))
                     else:
                         addValue(name, content.encode('hex'))
@@ -1229,6 +1233,16 @@ def process_all_mobi_headers(files, sect
             # if reach here should be an image but double check to make sure
             # Get the proper file extension
             imgtype = imghdr.what(None, data)
+            # imghdr only checks for JFIF or Exif JPEG files. Apparently, there are some with only the magic JPEG bytes out there...
+            # ImageMagick handle those, so, do it too.
+            if imgtype is None and data[0:2] == b'\xFF\xD8':
+                # Get last non-null bytes
+                last = len(data)
+                while (data[last-1:last] == b'\x00'):
+                    last-=1
+                # Be extra safe, check the trailing bytes, too.
+                if data[last-2:last] == b'\xFF\xD9':
+                    imgtype = "jpeg"
             if imgtype is None:
                 print "Warning: Section %s does not contain a recognised resource" % i
                 imgnames.append(None)

Last edited by NiLuJe; 09-19-2013 at 01:35 PM.
NiLuJe is offline   Reply With Quote
Old 09-20-2013, 09:45 AM   #577
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Your Patch and Page Maps

Hi,

Thanks for your patch. I already have the second part in the tree (your image fix) and will look at your new patch and include it in the next release (which should be soon since not much else seems to be happening).

I looked at the calibre patch you cited and instead of setting Offset to 0 if 0xffffffff perhaps we should simply not keep it in the metadata similar to what is being done in calibre?. I would guess a similar thing for the thumb offset.

Also on the subject of page maps. If anyone has a short epub with page information that I can play with to see exactly how Kindlegen generates its Page Map record, I should be able to figure out how to recreate the proper entries in the opf so that these do not get lost.

So if you have one of these I would love to get a copy so I can start working with it.

Thanks,

KevinH
KevinH is offline   Reply With Quote
Old 09-20-2013, 09:58 AM   #578
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 13,477
Karma: 26012492
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
@KevinH: Without the patch, it was blowing up later on somewhere during the opf build, I'd have to check how it behaves if we simply scrap the metadata field .

That said, I *think* setting it to 0 has the intended effect, and we at least don't magically make a field disappear [although, yeah, granted, it'd only pop up in the opf, I think] .

EDIT: If the fields are scrapped, well, they're scrapped, it doesn't blow up either, but, well, we lose them (no StartOffset anymore, probably not a big deal, but we do lose the cover thumbnail, and the cover image is not flagged as the cover in the OPF).

Last edited by NiLuJe; 09-20-2013 at 10:11 AM.
NiLuJe is offline   Reply With Quote
Old 09-20-2013, 06:32 PM   #579
clee
Member
clee began at the beginning.
 
Posts: 13
Karma: 37
Join Date: Mar 2009
Device: PRS-505
Not that I want to tell anybody here how they should manage their software, but man, it would really be convenient if KindleUnpack's history were on GitHub.
clee is offline   Reply With Quote
Old 09-20-2013, 08:12 PM   #580
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Quote:
Originally Posted by clee View Post
Not that I want to tell anybody here how they should manage their software, but man, it would really be convenient if KindleUnpack's history were on GitHub.
For a number of reasons
- we used to have a google code repository but it went basically unused
- the source is not big, is not that complicated, and does not change rapidly
- it is only released when new features or serious bug fixes are needed
- it is standalone and not part of another project
- it was derived from the work of a number of people but only a few of us maintain it and we are all active on this site (Mobileread)

Why is the history important? The original mobi 6 version can be found at the start of this thread and all releases of the dual mobi/KF8 version replace the earlier versions completely. The kindleunpack file has the version history at the top if you need to see what has changed in each release.


KevinH
KevinH is offline   Reply With Quote
Old 09-21-2013, 03:42 AM   #581
clee
Member
clee began at the beginning.
 
Posts: 13
Karma: 37
Join Date: Mar 2009
Device: PRS-505
Quote:
Originally Posted by KevinH View Post
For a number of reasons
- we used to have a google code repository but it went basically unused
- the source is not big, is not that complicated, and does not change rapidly
- it is only released when new features or serious bug fixes are needed
- it is standalone and not part of another project
- it was derived from the work of a number of people but only a few of us maintain it and we are all active on this site (Mobileread)

Why is the history important? The original mobi 6 version can be found at the start of this thread and all releases of the dual mobi/KF8 version replace the earlier versions completely. The kindleunpack file has the version history at the top if you need to see what has changed in each release.

KevinH
Back when it was still called mobiunpack, I ported some of the code to Ruby for use in some other software. As a developer who built some code based on an earlier version of the kindleunpack code, it would be really nice to see the actual code changes and bugfixes that went with each new version.

I'm not terribly surprised that the Google code repository didn't get used much; GitHub does make it a lot easier for new people to get involved and send pull requests (without even having to ever install any sort of SCM on their computer!).

If you still have archives of all of the old versions, I'd be happy to do some software archeology and set up a git repository with the history version-by-version! If not, or nobody else thinks that this would be of value, then I'll kindly go back to lurking.
clee is offline   Reply With Quote
Old 09-21-2013, 05:21 AM   #582
pdurrant
The Grand Mouse 高貴的老鼠
pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.
 
pdurrant's Avatar
 
Posts: 71,406
Karma: 305065800
Join Date: Jul 2007
Location: Norfolk, England
Device: Kindle Voyage
Quote:
Originally Posted by clee View Post
If you still have archives of all of the old versions, I'd be happy to do some software archeology and set up a git repository with the history version-by-version! If not, or nobody else thinks that this would be of value, then I'll kindly go back to lurking.
I have archives of almost everything. I suspect that Kevin and others also have code archives. I'd be happy to send you what I have. Send me a PM with an address.
pdurrant is offline   Reply With Quote
Old 09-22-2013, 06:27 PM   #583
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Quote:
Originally Posted by pdurrant View Post
I have archives of almost everything. I suspect that Kevin and others also have code archives. I'd be happy to send you what I have. Send me a PM with an address.
Hi,
Yes I have a bunch of previous versions someplace but many of mine might not match official releases as I was playing around with some changes at times.
I am sure together with Paul we can put together a reasonably complete set. I can be reached here by PMing KevinH
KevinH is offline   Reply With Quote
Old 09-25-2013, 10:07 PM   #584
Julius Caesar
Addict
Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.Julius Caesar ought to be getting tired of karma fortunes by now.
 
Posts: 341
Karma: 1244808
Join Date: Oct 2012
Device: Kindle Paperwhite
Quote:
Originally Posted by pdurrant View Post
Are you sure that all the page information is in your original Kindle file? Have you tried completely deleting the book from your device, along with all auxiliary files, and then putting the book back on the device (while keeping the device from connecting to the internet), to check that the page info is still found.
Quote:
Originally Posted by Doitsu View Post
Even then it wouldn't work, since page numbers are stored in a separate .apnx file that KindleGen cannot generate, and, AFAIK, even though the APNX format has been reverse-engineered by user_none, no tool exists that'll generate a separate .apnx file with accurate page numbers from an NCX pagemap or a mobi file that was compiled with an NCX pagelist in it.
(Calibre can only generate algorithm-based .apnx files with page numbers.)
Problem solved .

It turned out that the problem is not with Kindle Unpack but with Kindlegen. An eBook generated by Kindlegen contains 2 components, the mobi file and the KF8 file, aka azw3 file. Kindle can not see the page number when these two different formats are in the same package. So, the solution is to split them with Kindle Unpack and only load the KF8 file to the device.

Thanks to all contributors who made Kindle Unpack possible. This tool has been tremendously useful.

Last edited by Julius Caesar; 09-26-2013 at 02:42 AM.
Julius Caesar is offline   Reply With Quote
Old 10-08-2013, 07:27 AM   #585
tacitus
Member
tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.tacitus ought to be getting tired of karma fortunes by now.
 
Posts: 11
Karma: 288770
Join Date: Jun 2012
Device: none
I apologize if this has been answered in one of the previous 39 pages, but I'm using KindleUnpack to access a mobi file of my book generated by Calibre. The formatting of the mobi file is perfect for my needs, but the Calibre tool doesn't allow me to use the Table of Contents I added myself (the formatting doesn't allow the conversion tool to detect the TOC correctly).

All I want to do is add the requisite "toc" tag in the HTML file and add a line to the OPF so that the Goto feature works correctly for my book, which is why I was trying KindleUnpack (which works great). But after making the edits (just one line each in the HTML and OPF files), when I zip up the files and either use KindleGen or the online KDP tool, I lose a lot of the formatting of my book -- namely the spacing and indenting of all the paragraphs, the top margins, and any empty lines I use for padding at the top of the page.

It seems as though some of the requisite CSS is missing -- there are no stylesheet settings for paragraphs in the HTML anywhere in the html file, just the inline formatting on the tags themselves (which doesn't specify margins or indents).

I must be missing something? Calibre gets me so close, with the exception of this TOC issue, and with KindleUnpack I have a way to fix that, but the formatting then goes to pot.

It's very frustrating. (I've also tried going via epub but Sigil seemed to screw things up too!). Am I missing something, or is KindleUnpack not extracting something that would allow the Calibre formatting to be maintained (like a stylesheet, or something)?
tacitus is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Can i rotate text and insert images in Mobi and EPUB? JanGLi Kindle Formats 5 02-02-2013 04:16 PM
PDF to Mobi with text and images pocketsprocket Kindle Formats 7 05-21-2012 07:06 AM
Mobi files - images DWC Introduce Yourself 5 07-06-2011 01:43 AM
pdf to mobi... creating images rather than text Dumhed Calibre 5 11-06-2010 12:08 PM
Transfer of images on text files anirudh215 PDF 2 06-22-2009 09:28 AM


All times are GMT -4. The time now is 07:20 AM.


MobileRead.com is a privately owned, operated and funded community.