opf file size bug - slow opening ???

[cybmole]

this may not strictly be a calibre-only issue but hoping for some clues. esp on how calibre imports / generates .ofp files

I have one frustrating epub novel which takes 20 30 secs to open on my Sony PRS-650. I have spent ages trying to trouble shoot. have converted the epub to txt & back again to remove any junk formatting, have reformatted my sony memory, bug persists only with this one book....

Inspecting the associated opf file in calibre library I see that it contains a huge chunk of text - basically a whole chapter of the book, included in the summary as a free extract.. I'm wondering if that can / should have an impact on opening time. the book opens fine on PC in calibre viewer.

if that is not the answer then I am truly as a loss as to what could survive an ebub to txt to epub conversion & live on to cause trouble ???

it seems that is the cause though: having deleted a big chunk of the metadata in calibre book summary pane, the book is now opening OK on Sony.

any insights gratefully received. I think I can attach the original .opf without breaching any copyrights , so here it is :

PS could someone also clarify how the original .ofp is getting into calibre.

I have a source folder(outside of calibre) with epub, .ofp & jpg files;
I delete the book completely from calibre , then I copy only the source epub file to my desktop & then drag it into calibre, yet it appears in calibre library with a full set of metadata.
So is that stuff also embedded in epub or is calibre finding old data because book was previously in calibre library ?

PPS if I delete metadata then use calibre to search for replacement metadata it find this:

Vengeance Road
Rick Mofina
★★★★
See at: Google, Amazon.com, 9780778326380

and brings down that huge "free extract" of the book within the summary.

so is it "within spec" to have all that data in that file, and thus a ADE/Sony bug which causes the sony reader to choke on it, or is it a mis-use of metdata ?

[Manichean]

Moderator Notice
Attachment removed. Remember that copyrighted material cannot be uploaded to or attached to posts on Mobileread.

[DoctorOhh]

The opf file discussed here is the opf file found in the calibre library book folder and is not copyrighted. This file is created by calibre and used to restore the database in the event of catastrophic metadata.db failure.

This file also is not sent to the ereader when you send the epub, so this file shouldn't have anything to do with slowness of ereader opening the epub.

[cybmole]

Originally Posted by dwanthny

The opf file discussed here is the opf file found in the calibre library book folder and is not copyrighted. This file is created by calibre and used to restore the database in the event of catastrophic metadata.db failure.

This file also is not sent to the ereader when you send the epub, so this file shouldn't have anything to do with slowness of ereader opening the epub.

thanks for backing me up on point 1. the opf content which I uploaded is indeed what calibre downloads from public domian amazon.com when asked to fetch metadata , and so is NOT copyright


re point 2. surely the ofp is also part of the epub ( zip ) bundle and thus does get into the reader, and does get into calibre when only the .epub is imported.. if I unzip my source .epub I see a file called content.ofp in the bundle. this contains the same data as a calibre metadata download.
I have no other explanation
for epub with original (large) ofp = very slow to open; epub with edited ( short) ofp, opens normally.

it is not "normal" for downloaded metadata to contain an entire "sample chapter", I have no other instances of that, & no other books that take 30+secs to open!

so, having googled .opf files, I would guess that the delay is caused by reader software having to parse a whole HTML book chapter when analysing the large .opf. It's also maybe relevant that on sony, I can open book ( 30+secs), close it, then re-open it & it opens immediately the 2nd time ( opf analysis buffered ??? ) . But if I open another book in between I'm back to the v slow issue.

[DoctorOhh]

Originally Posted by cybmole

re point 2. is the ofp also part of the epub ( zip ) bundle and thus does get into the reader, and does get into calibre when only the .epub is imported.

The opf in calibre's library is different then the one found in the epub itself. That said, it is not surprising if you see that full excerpt listed in the metadata. I opened a book in Sigil to take a look and the metadata did have the summary.

Just edit the metadata in calibre and delete the chapter excerpt from the summary and resend the book to your device. This should remove the offending info from the books metadata.

[cybmole]

yes, that fixes the specific issue, but the more general question is who/ what was at fault here.

sony/ ADE software for not coping with a sample chapter embedded within .ofp; calibre for putting it there via metadata download; the publisher for putting it up on amazon....some combination of all 3 ???

i did google .ofp spec but found nothing specifically allowing or disallowing book text extracts.

PS it was also very frustrating - I investigated so many "wrong" answers before looking at .ofp. I converted the book into & out of .mobi, of .rtf, of .txt, I crawled through all the code & css in sigil looking for culprits, I reformatted my Sony drive even! - losing all my bookmarks in the process, thinking that is must be a hardware issue after converting to / from .txt & still seeing the bug!

[itimpi]

It think that Calibre is correct in including all the metadata that you set in Calibre into the .opf file. I do not see how Calibre can sensibly decide what metadata to include, so should include all it is given. Therefore the culprit is the person who supplied that inappropriate metadata in the first place (Amazon), or you for leaving it set in the Calibre metadata after downloading it? Of course it would also not be an issue if the reading software could handle it being present.

One thing that Calibre could perhaps do is provide a warning if it exceeds some arbitay limit, but what that limit should be I have no idea.

[cybmole]

thanks for those comments. it seems to be only the sony reader software that chokes. I have ADE on PC & that opens the original epub OK, as did calibre viewer. As Sony gives no helpful error messages there seems no way to dig any deeper.

[JSWolf]

Originally Posted by cybmole

thanks for those comments. it seems to be only the sony reader software that chokes. I have ADE on PC & that opens the original epub OK, as did calibre viewer. As Sony gives no helpful error messages there seems no way to dig any deeper.

Your computer is a lot more powerful then any reader. So of course, you won't see any slowdown with ADE on your computer.

[theducks]

Originally Posted by cybmole

thanks for those comments. it seems to be only the sony reader software that chokes. I have ADE on PC & that opens the original epub OK, as did calibre viewer. As Sony gives no helpful error messages there seems no way to dig any deeper.

How big (bytes) was that OPF?
I would expect that a OPF would have the same (segment) size constraints as the books content

[Manichean]

Originally Posted by dwanthny

The opf file discussed here is the opf file found in the calibre library book folder and is not copyrighted. This file is created by calibre and used to restore the database in the event of catastrophic metadata.db failure.

Ah, I may have missed the attachment type... cybmole, if you attached the OPF file, I apologize for deleting it. That's what happens if you just pop into the forum quickly between running lengthier calculations at work. Feel free to reattach the OPF if needed.

[cybmole]

its only 16k , not enough to trigger a page error.

the Sony does eventually cope, if you go away & make a cup of tea or something then come back a while later - but the apparency is that the reader has frozen & several times I resorted to pin into reset hole.... then hold power switch slider & pray.....

- copy of source .opf re-attached. ( it was a "found" book) & the .opf appears to be calibre-generated

one anomaly- please check this - if I open the attached file in notepad I see the big sample chapter html, but if I open in notepad++ I see only the short, useful meta stuff. no book text at all.

[theducks]

Originally Posted by cybmole

its only 16k , not enough to trigger a page error.

the Sony does eventually cope, if you go away & make a cup of tea or something then come back a while later - but the apparency is that the reader has frozen & several times I resorted to pin into reset hole.... then hold power switch slider & pray.....

- copy of source .opf re-attached. ( it was a "found" book) & the .opf appears to be calibre-generated

I don't know if the Huge (IMHO mangled HTML) description: section is the problem.

</p><p>

All the Paragraph tags have been converted to entities,making the description, one Looooong sentence

[cybmole]

Originally Posted by theducks

I don't know if the Huge (IMHO mangled HTML) description: section is the problem.

</p><p>

All the Paragraph tags have been converted to entities,making the description, one Looooong sentence

yes- the whole epub was like that, but it did'nt seem to upset the reader. I fixed all that for the book text with sigil regex, got it nicely formatted, then still had the slow opening bug. that's when I got into converting it to / from mobi, rtf, txt etc trying to figure what code could be causing the problem. I wasted a couple of hours on that, then another hour testing for possible problems with the reader before looking at the .ofp. I checked that I have latest sony firmware, reset everything to factory default, reformatted .... added/opened/deleted lots of other books OK...

I also have a sneaky feeling also that the book is missing 4 chapters towards the end; when I fixed up the TOC, it is missing 4 chapter numbers!

[jackie_w]

Originally Posted by cybmole

I also have a sneaky feeling also that the book is missing 4 chapters towards the end; when I fixed up the TOC, it is missing 4 chapter numbers!

Just FYI - the contents of the TOC (.ncx file) would not necessarily tell you whether any chapters were missing. The important thing is whether all the (x)html files for all chapters are present in the epub and whether all those html files are listed in both the <spine> section and the <manifest> section of the .opf file inside the epub.