Glo Database malformed but signout does not work...

Mrs_Often · 09-20-2015, 10:18 AM

Hello!

I have a few books on my Glo that for some reason don't show the cover. I found out for some of them that the cover image in the epub was not called "cover" but something like "img001" or something (I didn't look at all of them yet). I vaguely recalled from somewhere that Kobo likes covers to be called cover (Q1. is this [still] true?), so I changed the name, tested the epub on computer, and re-sent the book to the device with Calibre.

Didn't work. I also used Calibre KoboUtilities plugin to "Upload covers for selected books" but that didn't make the covers show up either. And then I noticed one of the books I had deleted and re-sent to the device was not showing up at all. Furthermore, one book that I had NOT deleted but re-sent anyway was showing up twice, but only one of them could be opened... So, database broken.

I tried signing out... Device asks me if I'm sure, I say yes, and it sits there saying "Signing out, please wait..." for a while. Then it restarts, and shows me my beautiful home screen without any change... plus I'm not signed out at all. I can repeat this step many times but I never really get to sign out.

This makes me happy because my number of books read in number of hours isn't lost. But... it makes me sad because my database is still broken.

Calibre gives me this message when connecting device:

Spoiler:

I also tried the option "Compress the device database" from the KoboUtility, not sure what exactly that does, but I tried it anyway, and Calibre tells me:

Spoiler:

Questions:
1. Does Kobo (still) want cover images to be called cover?
2. Any thoughts on database issue? I would be very happy if I wouldn't need to factory reset...
3. Also, is there any way, once I DO get to rebuild the database, to get my statistics (# books read/# hours) back? Can I manually enter them anywhere in the database?
3. I noticed in Calibre KoboUtility plugin under "Update Metadata in Device Library" that there is an option to change reading status... handy for the books I have read, to change them to "Finished" after database rebuilds... but... how do I tell it which books to set to "Finished"? I can't seem to find that info in the Help section, and I don't want to risk changing all the books to "Finished" by trying the option without being sure

Thanks!

Deahna · 09-20-2015, 01:12 PM

Quote:

Originally Posted by Mrs_Often

3. I noticed in Calibre KoboUtility plugin under "Update Metadata in Device Library" that there is an option to change reading status... handy for the books I have read, to change them to "Finished" after database rebuilds... but... how do I tell it which books to set to "Finished"? I can't seem to find that info in the Help section, and I don't want to risk changing all the books to "Finished" by trying the option without being sure

Thanks!

This, at least, I can help with.

Change the view in calibre so it shows you the books on your device. Then you select those you want to change to "finished" and then you use Kobo Utilities. When done it will tell you how many books were selected, changed, unchanged...

I tried first with only one book (just in case!) and it worked perfectly on both FW 3.15 and 3.17.3.

Unfortunately it's not possible to select the books in the library view, you'll have to track them down in the device view one by one. Next time I have to factory reset a kobo I will first copy all finished books to the device (if I want them there), change the status and then add the rest.

Mrs_Often · 09-20-2015, 01:47 PM

Thank you Deahna!

GeoffR · 09-20-2015, 02:06 PM

Which firmware version is the Glo running? Are you sideloading ePubs or KePubs?

The Glo has never needed the cover image to have any special name in ePubs, but I don't know about kePubs. Perhaps Calibre itself might use the filename to decide which to use as cover if it can't work it out another way though.

If signing out worked then it would wipe the database and you would lose all your stats for sideloaded books anyway. A factory reset also reformats the filesystem and so you also lose the sideloaded books themselves and have to reload them, but that should not be much trouble if they are all in Calibre.

I don't know what could cause signing out to fail, the only possibility that comes to mind is if you are running firmware 3.16.10 or 3.17.0 then the over-zealous sickel process might cause the device to reboot if the sign-out procedure was taking too long.

You might want to wait to hear from davidfor or someone with ideas about what you could do to recover the reading stats of sideloaded books from the database. But I don't think it is a simple process unless you already have a recent backup of the database from before it was corrupted.

DNSB · 09-20-2015, 02:59 PM

Quote:

Originally Posted by Mrs_Often

I tried signing out... Device asks me if I'm sure, I say yes, and it sits there saying "Signing out, please wait..." for a while. Then it restarts, and shows me my beautiful home screen without any change... plus I'm not signed out at all. I can repeat this step many times but I never really get to sign out.

This makes me happy because my number of books read in number of hours isn't lost. But... it makes me sad because my database is still broken.

Unhappily, I'd suspect that you may need to do a factory reset. Short of your database being flagged as read-only, I really don't know why the signout would not replace it with a fresh copy.

Quote:

Originally Posted by GeoffR

Which firmware version is the Glo running? Are you sideloading ePubs or KePubs?

The Glo has never needed the cover image to have any special name in ePubs, but I don't know about kePubs. Perhaps Calibre itself might use the filename to decide which to use as cover if it can't work it out another way though.

The difference is that a KePub needs to have the following in the content.opf:

Code:

 <item href="Images/cover.jpg" id="cover-image" media-type="image/jpeg" properties="cover-image"/>

whereas an ePub v2 uses:

Code:

<item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" />

I've never done any testing without one of those entries so can't say whether the Kobo firmware will find the cover if one of those entries is not present. As long as the filename and extension are correct (assuming the file and extension are a supported image format), it doesn't seem to matter what the name is.

Mrs_Often · 09-20-2015, 03:47 PM

Quote:

Originally Posted by GeoffR

Which firmware version is the Glo running? Are you sideloading ePubs or KePubs?

3.17.3, sideloading KePubs

Quote:

Originally Posted by GeoffR

If signing out worked then it would wipe the database and you would lose all your stats for sideloaded books anyway.

Are you sure? I thought all sideloaded books would stay on the device but the database would be rebuilt (indeed losing stats)... If you're correct then indeed I might as well do a factory reset.

Quote:

Originally Posted by GeoffR

I don't know what could cause signing out to fail, the only possibility that comes to mind is if you are running firmware 3.16.10 or 3.17.0 then the over-zealous sickel process might cause the device to reboot if the sign-out procedure was taking too long.

Ahh very interesting! The "Signing out, please wait" is indeed taking quite a while before the device restarts... But, I have 3.17.3, not 3.17.0... hmmm

Quote:

Originally Posted by GeoffR

You might want to wait to hear from davidfor or someone with ideas about what you could do to recover the reading stats of sideloaded books from the database. But I don't think it is a simple process unless you already have a recent backup of the database from before it was corrupted.

Yes I'm hoping to hear from him indeed

Unfortunately all the backups I have are either from too long ago or already have corrupted database

Quote:

Originally Posted by DNSB

Unhappily, I'd suspect that you may need to do a factory reset. Short of your database being flagged as read-only, I really don't know why the signout would not replace it with a fresh copy.

I was afraid someone might dare say that

Hmm read-only... also an interesting thought. As far as I can see from my Mac OS X terminal, the .sqlite permissions appear to be read/write/executable...

Quote:

Originally Posted by DNSB

The difference is that a KePub needs to have the following in the content.opf:

Code:

<item href="Images/cover.jpg" id="cover-image" media-type="image/jpeg" properties="cover-image"/>

whereas an ePub v2 uses:

Code:

<item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" />

[...] it doesn't seem to matter what the name is.

Is the only difference then the ID format and the extra properties? I'll go back and check the books that aren't showing any cover, see if those are indeed the difference with the books that are

Thank you both for your input!

Mrs_Often · 09-20-2015, 04:00 PM

Hmm...

From content.opf of a sideloaded KePub that does NOT show the cover:

Code:

<item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" />

From content.opf of a sideloaded KePub that DOES show the cover:

Code:

 <item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" />

So no difference there. And the href's are indeed referring to the correct location

GeoffR · 09-20-2015, 04:22 PM

Quote:

Originally Posted by Mrs_Often

Are you sure? I thought all sideloaded books would stay on the device but the database would be rebuilt (indeed losing stats)... If you're correct then indeed I might as well do a factory reset.

That is actually what I said -- Signing out just wipes the database, a factory reset wipes the database and sideloaded books. But reloading the books is usually the easy part, recovering the stats from the database is the difficult part (if possible at all), so as long as you have copies of all your books in Calibre then a factory reset won't lose any more information than just signing out would have.

Mrs_Often · 09-20-2015, 04:36 PM

My super sexy husband fixed my database

Apparently Macs have a built in sqlite command line tool and husband was able to dump database to a text file and recreate it from there

He also found a couple of culprit books... No idea if I'm able to sign out now, but I don't really want to try

So now I can get to puzzling on why some book covers aren't showing. Any idea on whether I need to delete a book with Calibre before re-sending it to the device if I made any changes with Sigil? Edit: apparently not, same thing I tried before worked perfectly this time now the database doesn't have its knickers in a twist

Also still interested to know about question 3, for future reference or for when I'd like to cheat my statistics...

Mrs_Often · 09-20-2015, 04:38 PM

Quote:

Originally Posted by GeoffR

That is actually what I said -- Signing out just wipes the database, a factory reset wipes the database and sideloaded books. But reloading the books is usually the easy part, recovering the stats from the database is the difficult part (if possible at all), so as long as you have copies of all your books in Calibre then a factory reset won't lose any more information than just signing out would have.

Aaah right sorry, I misread a "for" as an "and", whoops

There's me practising speed reading at the wrong time

GeoffR · 09-20-2015, 05:15 PM

Quote:

Originally Posted by Mrs_Often

Any idea on whether I need to delete a book with Calibre before re-sending it to the device if I made any changes with Sigil? Edit: apparently not, same thing I tried before worked perfectly this time now the database doesn't have its knickers in a twist

It depends on what changes you make. If you change the file structure of the book (add, rename or delete files) or if you change parts the OPF or NCX or the places in the book that are linked to in the NCX then you should delete the book before reloading it, otherwise the database will still have the info from the old book which won't match the new book. This can cause unpredictable effects while trying to navigate the book.

If you just change the content of the book, fixing spellng mistakes or replacing images (without changng their filenames) etc., then it is okay to resend the updated book to the device without deleting it first.

Quote:

Also still interested to know about question 3, for future reference or for when I'd like to cheat my statistics...

I don't think there are any running totals stored in the database. When you open the stats page it seems to trawl through the database to compile the stats from the raw data on each reading session, which is why it can take a long time to bring up the stats page the first time after rebooting the device. After that it might keep running totals in memory, but they don't appear to be saved anywhere when the device is powered off.

So in short, it is probably possible to fake the statistics, but it would be a lot of effort, I think you would have to insert a lot of fake reading sessions into the database.

koboy · 09-20-2015, 05:24 PM

Quote:

Originally Posted by Mrs_Often

My super sexy husband fixed my database

Apparently Macs have a built in sqlite command line tool and husband was able to dump database to a text file and recreate it from there

He also found a couple of culprit books... No idea if I'm able to sign out now, but I don't really want to try

So now I can get to puzzling on why some book covers aren't showing. Any idea on whether I need to delete a book with Calibre before re-sending it to the device if I made any changes with Sigil? Edit: apparently not, same thing I tried before worked perfectly this time now the database doesn't have its knickers in a twist

Also still interested to know about question 3, for future reference or for when I'd like to cheat my statistics...

regarding the book covers, do you have more than 100mb of free space on the Glo? if not delete some books until you have at least 150mb spare, and then see if the book covers are showing.

best wishes koboy

davidfor · 09-20-2015, 10:36 PM

Quote:

Originally Posted by Mrs_Often

Hmm...

From content.opf of a sideloaded KePub that does NOT show the cover:

Code:

<item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" />

From content.opf of a sideloaded KePub that DOES show the cover:

Code:

 <item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" />

So no difference there. And the href's are indeed referring to the correct location

For the firmware to get the kepub cover, the important bit is the "properties". The id has to there, but it is used as a cross-reference to other entries in the OPF. It doesn't matter what the value is.

If you have kepubs without the properties for the cover image that are showing covers, the covers probably got there some other way. Either the driver has been configured to send the covers or you used the Kobo Utilities to send them.

davidfor · 09-20-2015, 10:38 PM

Quote:

Originally Posted by koboy

regarding the book covers, do you have more than 100mb of free space on the Glo? if not delete some books until you have at least 150mb spare, and then see if the book covers are showing.

I doubt this is the case. I have deliberately filled the device to test a couple of things. I had no problems except if a firmware update was available.

davidfor · 09-20-2015, 10:59 PM

Quote:

Originally Posted by Mrs_Often

My super sexy husband fixed my database

Apparently Macs have a built in sqlite command line tool and husband was able to dump database to a text file and recreate it from there

If he did a check on the database and kept the list of errors, I'd be interested in seeing them. I have fixed databases in the past, but with very limited type of errors.

Quote:

He also found a couple of culprit books... No idea if I'm able to sign out now, but I don't really want to try

So now I can get to puzzling on why some book covers aren't showing. Any idea on whether I need to delete a book with Calibre before re-sending it to the device if I made any changes with Sigil? Edit: apparently not, same thing I tried before worked perfectly this time now the database doesn't have its knickers in a twist

It really depends on when the cover didn't show up. If it never did, then it probably means the driver couldn't identify the cover and set the property when the book was sent. If it disappeared after a while, I don't know. There is a discussion on this in the extended driver thread with no conclusion. I did have some problems with firmware 3.16.10 and 3.17.0 that meant the stored cover image file was empty. In these cases, the cover was not displayed after I restarted the device. Removing the files and restarting the device generated correct covers. I think I have seen disappearing covers with 3.17.3, but haven't had one since the discussion started.

Quote:

Also still interested to know about question 3, for future reference or for when I'd like to cheat my statistics...

I can't help with getting the reading stats back after a factory reset. Some will come back from the Kobo server if syncs happen, but not all. For marking the books as read, just select the books you have read in your calibre library and update the device using the mentioned Kobo Utilities plugin. Of course, that needs an indication in the library. I put all the read books in a reading list. And in the past, I had a Yes/No column that I set to yes when I read the book. Of course, nowadays I let the utilities plugin do all the hard work.

The "Store/Restore current bookmark" will store the current reading status of the books on the device in the calibre library. After a factory reset or logout/in, you can restore the status. The status includes the actual reading position, percent read, when you last read it and the rating (though this isn't in the latest firmware for sideloaded books). The store can be automated so that it happens each time the device is seen by calibre.

Another thing you can do is backup the database. Then, if it gets corrupted, you restore the last good copy and you don't lose as much history. The utilities plugin can automate this as well.

09-20-2015, 10:18 AM	#1
Mrs_Often Wizard Posts: 1,775 Karma: 2694823 Join Date: Dec 2011 Location: The Netherlands Device: Kobo Touch, Glo, Clara HD	Database malformed but signout does not work... Hello! I have a few books on my Glo that for some reason don't show the cover. I found out for some of them that the cover image in the epub was not called "cover" but something like "img001" or something (I didn't look at all of them yet). I vaguely recalled from somewhere that Kobo likes covers to be called cover (Q1. is this [still] true?), so I changed the name, tested the epub on computer, and re-sent the book to the device with Calibre. Didn't work. I also used Calibre KoboUtilities plugin to "Upload covers for selected books" but that didn't make the covers show up either. And then I noticed one of the books I had deleted and re-sent to the device was not showing up at all. Furthermore, one book that I had NOT deleted but re-sent anyway was showing up twice, but only one of them could be opened... So, database broken. I tried signing out... Device asks me if I'm sure, I say yes, and it sits there saying "Signing out, please wait..." for a while. Then it restarts, and shows me my beautiful home screen without any change... plus I'm not signed out at all. I can repeat this step many times but I never really get to sign out. This makes me happy because my number of books read in number of hours isn't lost. But... it makes me sad because my database is still broken. Calibre gives me this message when connecting device: Spoiler: calibre, version 2.38.0 ERROR: Error: Error communicating with device database disk image is malformed Traceback (most recent call last): File "site-packages/calibre/gui2/device.py", line 87, in run File "site-packages/calibre/gui2/device.py", line 533, in _sync_booklists File "calibre_plugins.kobotouch_extended.device.driver" , line 331, in sync_booklists File "calibre_plugins.kobotouch_extended.device.driver" , line 327, in __rows_needing_imageid DatabaseError: database disk image is malformed I also tried the option "Compress the device database" from the KoboUtility, not sure what exactly that does, but I tried it anyway, and Calibre tells me: Spoiler: calibre, version 2.38.0 ERROR: Unhandled exception: <b>DatabaseError</b>:database disk image is malformed calibre 2.38 isfrozen: True is64bit: True Darwin-13.4.0-x86_64-i386-64bit Darwin ('64bit', '') ('Darwin', '13.4.0', 'Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64') Python 2.7.9 OSX: ('10.9.5', ('', '', ''), 'x86_64') Successfully initialized third party plugins: K4PC, K4Mac, Kindle Mobi and Topaz DeDRM && Inept Epub DeDRM && Open With && Goodreads && Ignoble Epub DeDRM && KoboBooks && Resize Cover && KePub Output && eReader PDB 2 PML && Inept PDF DeDRM && EpubSplit && Kindle Collections && Modify ePub && KoboTouchExtended && Kobo Utilities Traceback (most recent call last): File "calibre_plugins.koboutilities.action", line 1441, in vacuum_device_database File "calibre_plugins.koboutilities.action", line 3189, in _vacuum_device_database DatabaseError: database disk image is malformed Questions: 1. Does Kobo (still) want cover images to be called cover? 2. Any thoughts on database issue? I would be very happy if I wouldn't need to factory reset... 3. Also, is there any way, once I DO get to rebuild the database, to get my statistics (# books read/# hours) back? Can I manually enter them anywhere in the database? 3. I noticed in Calibre KoboUtility plugin under "Update Metadata in Device Library" that there is an option to change reading status... handy for the books I have read, to change them to "Finished" after database rebuilds... but... how do I tell it which books to set to "Finished"? I can't seem to find that info in the Help section, and I don't want to risk changing all the books to "Finished" by trying the option without being sure Thanks!

09-20-2015, 04:00 PM	#7
Mrs_Often Wizard Posts: 1,775 Karma: 2694823 Join Date: Dec 2011 Location: The Netherlands Device: Kobo Touch, Glo, Clara HD	Hmm... From content.opf of a sideloaded KePub that does NOT show the cover: Code: <item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" /> From content.opf of a sideloaded KePub that DOES show the cover: Code: <item href="Images/cover.jpg" id="cover.jpg" media-type="image/jpeg" /> So no difference there. And the href's are indeed referring to the correct location

09-20-2015, 04:36 PM	#9
Mrs_Often Wizard Posts: 1,775 Karma: 2694823 Join Date: Dec 2011 Location: The Netherlands Device: Kobo Touch, Glo, Clara HD	Woohoo My super sexy husband fixed my database Apparently Macs have a built in sqlite command line tool and husband was able to dump database to a text file and recreate it from there He also found a couple of culprit books... No idea if I'm able to sign out now, but I don't really want to try So now I can get to puzzling on why some book covers aren't showing. Any idea on whether I need to delete a book with Calibre before re-sending it to the device if I made any changes with Sigil? Edit: apparently not, same thing I tried before worked perfectly this time now the database doesn't have its knickers in a twist Also still interested to know about question 3, for future reference or for when I'd like to cheat my statistics... Last edited by Mrs_Often; 09-20-2015 at 04:48 PM. Reason: Too many annoying emoticons

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Aura H2O Koreader and malformed nickel database	Patricius	Kobo Reader	38	08-03-2015 05:45 AM
Strange malformed database error	itimpi	Library Management	2	12-19-2013 09:00 AM
Database disk image is malformed	moonsjine	Library Management	1	06-16-2012 04:03 AM
help: 'malformed database' with kobo	prairiegal	Devices	2	03-10-2011 09:50 PM
Malformed Database Request	timlegge	Kobo Reader	0	02-18-2011 10:44 PM

09-20-2015, 01:47 PM	#3
Mrs_Often Wizard Posts: 1,775 Karma: 2694823 Join Date: Dec 2011 Location: The Netherlands Device: Kobo Touch, Glo, Clara HD	Thank you Deahna!

09-20-2015, 02:06 PM	#4
GeoffR Wizard Posts: 3,821 Karma: 19162882 Join Date: Nov 2012 Location: Te Riu-a-Māui Device: Kobo Glo	Which firmware version is the Glo running? Are you sideloading ePubs or KePubs? The Glo has never needed the cover image to have any special name in ePubs, but I don't know about kePubs. Perhaps Calibre itself might use the filename to decide which to use as cover if it can't work it out another way though. If signing out worked then it would wipe the database and you would lose all your stats for sideloaded books anyway. A factory reset also reformats the filesystem and so you also lose the sideloaded books themselves and have to reload them, but that should not be much trouble if they are all in Calibre. I don't know what could cause signing out to fail, the only possibility that comes to mind is if you are running firmware 3.16.10 or 3.17.0 then the over-zealous sickel process might cause the device to reboot if the sign-out procedure was taking too long. You might want to wait to hear from davidfor or someone with ideas about what you could do to recover the reading stats of sideloaded books from the database. But I don't think it is a simple process unless you already have a recent backup of the database from before it was corrupted.