Kobo Forma database problems

[DNSB]

Originally Posted by frostschutz

Well, it's a firmware issue either way. If there is a problem with a book, the worst that should happen is a message pop up "sorry, I could not handle book <insert filename here>". If the device hangs and reboots, that makes it a firmware issue by default.

Ummm... in a perfect world. In the real world, when a Kobo device finds an error that prevents importing, most often, it crashes silently. No restart and until you notice the missing books, no notice that import failed though I have seen a few stack_xx.log files. In that perfect world, the suppliers of the ebooks would have checked them and corrected any issues before making the ebook available.

My procedure is to check all my epubs with FlightCrew and epubcheck (epub2) or epubcheck (epub3) and correcting all errors before sending the epubs to my ereader. In the case of my Clara HD, after upgrading the internal storage to 32GB, I copied over 9214 epubs in one swell foop which took a while and then watched the import slowly crawl from 6% to 94% (the import process seems to start at 6% and end at 94% ). Reconnected to my laptop and update the metadata to get series, etc. added to the database. No errors and all the books were in the database with the correct information.

[kobayashi]

Loading a few books doesn't work, loading a few and deleting them doesn't work either. It's not a single book that's causing the issue.

Is there a way to stop the Forma from auto-updating when going through the initial setup process to test book loading on the original firmware?

[bongorash]

ok so have done some further testing, fortunately a friend of mine has just received a new forma and he agreed to let me set it up and see if I could duplicate the problem on his - I did !! This suggests to me that it is indeed a problem with one of the books in my library. Loaded about 200 books fine then loaded another 50 and got the reboot at 6% error, after numerous resets I think I have narrowed it down to 1 (or More) of 10 books causing the problem. Just now testing the sending of the books individually to narrow it down. Will report back soon.

[anacreon]

I transfer via Calibre and use the extension KoboTouchExtended to convert to kepubs while transferring. I've learned to check the "silently ignore failed transfers", and transferring in small batches, then splitting the batches that failed. And at the end I manage to transfer everything, though it is long and painful (I have some 4000 books on the Forma). I wonder whether there is a way to do something similar with your problem.

[davidfor]

Originally Posted by bongorash

ok so have done some further testing, fortunately a friend of mine has just received a new forma and he agreed to let me set it up and see if I could duplicate the problem on his - I did !! This suggests to me that it is indeed a problem with one of the books in my library. Loaded about 200 books fine then loaded another 50 and got the reboot at 6% error, after numerous resets I think I have narrowed it down to 1 (or More) of 10 books causing the problem. Just now testing the sending of the books individually to narrow it down. Will report back soon.

If you work out which book is the problem, I'd like to see it. I know this is an issue, but I haven't see it happen for a very long time. I'd like the chance to examine a book that triggers it to work out what it is that causes it.

[bongorash]

So after much time copying files over in small batches I have now established that the problems have been caused by one 'rogue' book. I have loaded my entire library (some 500+ books) minus the 'rogue' book onto both mine and my friends Forma and everything went smoothly and all books copied fine. (I use calibre to side load them as kepubs). The strange thing is the rogue book loaded fine onto my Aura One so I can only think that the firmware for the Forma is more particular about which books it will load ?

davidfor:
I will pm you with a dropbox link to the offending file as you requested.

Although I have used calibre regularly I am not an expert on all the options available within it - is there any way that calibre can check a book for compatibility before I send it to my device ? This experience has shown me that when an ereader crashes over loading a rogue book it will not load any further books and a factory reset is necessary meaning reloading your full library again, not something I want to do regularly !

[kobayashi]

Originally Posted by anacreon

I transfer via Calibre and use the extension KoboTouchExtended to convert to kepubs while transferring. I've learned to check the "silently ignore failed transfers", and transferring in small batches, then splitting the batches that failed. And at the end I manage to transfer everything, though it is long and painful (I have some 4000 books on the Forma). I wonder whether there is a way to do something similar with your problem.

This kind of worked for me. Converting to kepubs got past the 6% hang but some books weren't formatted correctly and would have only a few sentences on each page.

[davidfor]

Originally Posted by bongorash

So after much time copying files over in small batches I have now established that the problems have been caused by one 'rogue' book. I have loaded my entire library (some 500+ books) minus the 'rogue' book onto both mine and my friends Forma and everything went smoothly and all books copied fine. (I use calibre to side load them as kepubs). The strange thing is the rogue book loaded fine onto my Aura One so I can only think that the firmware for the Forma is more particular about which books it will load ?

davidfor:
I will pm you with a dropbox link to the offending file as you requested.

Although I have used calibre regularly I am not an expert on all the options available within it - is there any way that calibre can check a book for compatibility before I send it to my device ? This experience has shown me that when an ereader crashes over loading a rogue book it will not load any further books and a factory reset is necessary meaning reloading your full library again, not something I want to do regularly !

I have had a look at the book, and the problem was easy to find. I dropped it on my Clara HD as an epub and it loaded OK. Then I simply renamed as a kepub, and the device rebooted during the import.

Opening it in the calibre editor and running the check there, it shows a lot of CSS errors. But also:

OEBPS/Content.opf (line: 18)
The meta cover tag points to an item with id="images/pic_1.jpg" which does not exist in the manifest

The line in the OPF was:

<meta name="cover" content="images/pic_1.jpg"/>

The value of the content attribute should be an id defined in the manifest part of the OPF. This should have just been "pic_1.jpg" as the manifest has the entry:

<item id="pic_1.jpg" href="images/pic_1.jpg" media-type="image/jpeg"/>

The way I fixed this was to right-click on the cover in the file list and select the option to mark it as a cover. That rewrote the first line above.

I tried the kepub version on my Touch which has firmware version 4.10.11586. The book imported OK there, but didn't have a cover. Look like there was a change in how kepubs are handled that breaks on this error.

[kobayashi]

I can confirm on my Forma that epubs are what's causing the issue. When converting all to kepub before transferring, they all load correctly on the Forma but some of the formatting is messed up.

The firmware on the Forma must be much less tolerant of any issues than all previous Kobo firmware.

[DNSB]

Originally Posted by davidfor

I have had a look at the book, and the problem was easy to find. I dropped it on my Clara HD as an epub and it loaded OK. Then I simply renamed as a kepub, and the device rebooted during the import.

Look like there was a change in how kepubs are handled that breaks on this error.

Originally Posted by kobayashi

I can confirm on my Forma that epubs are what's causing the issue. When converting all to kepub before transferring, they all load correctly on the Forma but some of the formatting is messed up.

It looks as if more than one issue is involved. Davidfor found that the epub loaded but the kepub rebooted his Clara HD while Kobayashi found the epub crashed but the kepub loaded though with odd formatting.

@kobayashi: Is it possible for you to PM either davidfor or myself with a sample epub that caused issues but converted to a kepub loaded without causing any visible problems?

[kobayashi]

Originally Posted by DNSB

@kobayashi: Is it possible for you to PM either davidfor or myself with a sample epub that caused issues but converted to a kepub loaded without causing any visible problems?

It's impossible to determine which epubs are causing issues. Sometimes a few will load without issue and another time those same epubs won't work. The entire factory reboot takes forever as well. Too bad there isn't a way to try the firmware that's already loaded onto the Forma without having to download the new one.

[DNSB]

Originally Posted by kobayashi

It's impossible to determine which epubs are causing issues. Sometimes a few will load without issue and another time those same epubs won't work. The entire factory reboot takes forever as well. Too bad there isn't a way to try the firmware that's already loaded onto the Forma without having to download the new one.

After the factory reset, connect the Forma to your computer and use the Kobo Desktop app to handle the setup and update. Once the app has finished, do not eject your Forma. Instead open it's exposed storage and delete the upgrade directory and the KoboRoot.tgz and manifest.md5sum files from the .kobo directory. After this do the eject. This should leave you on the factory firmware as long as you don't do a sync (turn off WiFi).

[davidfor]

Originally Posted by kobayashi

I can confirm on my Forma that epubs are what's causing the issue.

I have absolutely no doubt that some epubs cause problems like this. All I have said is that in this case, this is what is causing the problem. But, I wasn't at all surprised that that particular error in the epub didn't cause a problem. I know that the firmware doesn't look at that attribute for epubs. But, based on comments in the calibre forum, that error might cause a problem on other devices.

When converting all to kepub before transferring, they all load correctly on the Forma but some of the formatting is messed up.

Did you actually do a conversion to kepub? Or are you using the KoboTouchExtended driver to transform the epubs to kepubs? These work in very different ways. If it was an actual conversion, then I would not expect an error. A suggestion of how to fix these sort of problems is to do an epub-to-epub conversion. That will fix most errors in an epub. And an epub-to-kepub conversion is basically an epub-to-epub conversion plus adding the spans. Doing this would have definitely fixed this error.

The firmware on the Forma must be much less tolerant of any issues than all previous Kobo firmware.

Yes, they have changed something and missed catching an error. I have reported the problem pointing at this thread. Hopefully it will be fixed in the next update.

But, as a note, sideloaded kepubs aren't really a supported format. The only reference to them is as a suggestion for book creators to test their books before submitting them to Kobo. Kobo could come back and say "unsupported, fix the book first". I don't expect them to, but, it is a valid response from them.

[davidfor]

Originally Posted by kobayashi

It's impossible to determine which epubs are causing issues. Sometimes a few will load without issue and another time those same epubs won't work.

There might be some issues with the combinations of books. There might be a memory leak or something that means that a sequence of books will fail when any one of those books imported by itself would be OK. I can't judge easily as my experience is similar to that reported by DNSB. I have loaded thousands of books at a time, but, these are clean and I know they don't have any errors in them. I can only repeat the offer I made @bongorash, if someone finds a book that seems to be causing problems, then I'd love to look at it and try and work out why. And your comment might mean it has to extend to a set of books causing the problem.

[rashkae]

Originally Posted by kobayashi

Is there a way to stop the Forma from auto-updating when going through the initial setup process to test book loading on the original firmware?

Kind of... you can bypass registration, (lots of threads cover this this many times over the years.) The easy way is to just copy over a database file with a user record.. you're welcome to use this one:

https://www.transfer.sh/mfgKo/KoboReader.sqlite

But since you've already found that converting the books does't work without messing them up, that's pretty good indications that your books are fundamentally flawed.. and you've ignored multiple requests by the local experts to let them examine them, so I don't know where else you expect this thread to go.