Redundant/Invalid TOC entries

Stinger · 06-20-2010, 05:39 PM

So I think I've figured out the reason behind the TOC on the Kobo displaying redundant and/or invalid entries even though the epub TOC is fine and doesn't show any trace of these entries on other viewers/editors/devices.

I'm pretty sure it's tied in to the fact that the Kobo doesn't clean up when you delete a regular epub file from the device (leaves behind those residual image files, and also leaves behind ghost entries in the SQLite dbase).

When you delete the epub file from the Kobo, it apparantly leaves behind the TOC it cached in the SQLite dbase. If you re-send that same file to the Kobo, when it reprocesses, it seems to just add to those old entries.

I wiped my SQLite database just now, and reloaded a bunch of my books. I noticed that the TOC in those files is displayed perfectly fine now, with no redundant or old/invalid entries. Before the SQLite wipe, I know for a fact that these exact same epubs were showing a garbage TOC of 100+ items (caused by me playing with formatting and deleting and resending the epub to the Kobo multiple times).

I know people have asked about this a few times here, so at least we know the cause now (and a dirty solution).

pholy · 06-20-2010, 07:11 PM

I wonder if cleaning up the database is part of the new release, or if one of us will have to write a db app to clean up deleted files. As I recall, it shouldn't be too hard to do.

PeterT · 06-20-2010, 08:19 PM

It might be handy to have an "audit" program; one that would compare what was in the database to what was in the file system.

I know I saw some weirdness on my wife's Kobo; attaching it to a PC gave a few error messages about seeking past EOF in an ePub, plus, ADE shower errors on some books. I'd drop them through ADE, eject and reattach, and would get the same errors,

Finally looked at the drive by hand and saw some 0 length ePubs and other bits like that.

Dropped them by hand, cleaned up the images directory, and now all seems fine.

corona · 06-21-2010, 08:11 PM

I have noticed this too, without taking the time to really figure it out. Let's hope the firmware folks are thinking about this. I'm totally talking through my hat here, but it strikes me as a relatively uncomplicated problem.

timlegge · 06-26-2010, 09:02 PM

I was looking at the database tonight and wrote some queries that look for problem entries in the database. I suspect that any rows returned can be deleted but I have not tested it so you are on your own.

Looks for entries in the Content table that are not associated with a book entry

Code:

select * from Content where BookID not in (select contentid from content where BookID is Null )

Looks for entries in volume_shortcovers that have no related books

Code:

select * from volume_shortcovers where volumeid not in (select contentid from content where BookID is Null )

Tim

06-20-2010, 05:39 PM	#1
Stinger Asha'man Posts: 335 Karma: 844 Join Date: May 2010 Location: Canada Device: Kobo	Redundant/Invalid TOC entries So I think I've figured out the reason behind the TOC on the Kobo displaying redundant and/or invalid entries even though the epub TOC is fine and doesn't show any trace of these entries on other viewers/editors/devices. I'm pretty sure it's tied in to the fact that the Kobo doesn't clean up when you delete a regular epub file from the device (leaves behind those residual image files, and also leaves behind ghost entries in the SQLite dbase). When you delete the epub file from the Kobo, it apparantly leaves behind the TOC it cached in the SQLite dbase. If you re-send that same file to the Kobo, when it reprocesses, it seems to just add to those old entries. I wiped my SQLite database just now, and reloaded a bunch of my books. I noticed that the TOC in those files is displayed perfectly fine now, with no redundant or old/invalid entries. Before the SQLite wipe, I know for a fact that these exact same epubs were showing a garbage TOC of 100+ items (caused by me playing with formatting and deleting and resending the epub to the Kobo multiple times). I know people have asked about this a few times here, so at least we know the cause now (and a dirty solution).

06-26-2010, 09:02 PM	#5
timlegge Addict Posts: 305 Karma: 78651 Join Date: May 2010 Location: New Brunswick, Canada Device: Kobo, Kobo Wifi, Kobo Touch	I was looking at the database tonight and wrote some queries that look for problem entries in the database. I suspect that any rows returned can be deleted but I have not tested it so you are on your own. Looks for entries in the Content table that are not associated with a book entry Code: select * from Content where BookID not in (select contentid from content where BookID is Null ) Looks for entries in volume_shortcovers that have no related books Code: select * from volume_shortcovers where volumeid not in (select contentid from content where BookID is Null ) Tim

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
TOC Entries from PDF	HamsterRage	Calibre	2	07-12-2010 11:57 AM
Generating TOC entries for prelims / end matter in InDesign	forlor	ePub	7	07-07-2010 08:26 AM
patch: LrfError: page id invalid in toc	grimborg	Calibre	0	04-07-2010 05:22 AM
Book Designer / LRF - How many entries in TOC	Bilbo1967	Workshop	2	04-12-2009 10:20 AM
Best way to do a TOC for large numbers of entries?	HarryT	Workshop	5	01-18-2009 07:02 AM

06-20-2010, 07:11 PM	#2
pholy Booklegger Posts: 1,801 Karma: 7999816 Join Date: Jun 2009 Location: Toronto, Ontario, Canada Device: BeBook(1 & 2010), PEZ, PRS-505, Kobo BT, PRS-T1, Playbook, Kobo Touch	I wonder if cleaning up the database is part of the new release, or if one of us will have to write a db app to clean up deleted files. As I recall, it shouldn't be too hard to do.

06-20-2010, 08:19 PM	#3
PeterT Grand Sorcerer Posts: 13,270 Karma: 78869092 Join Date: Nov 2007 Location: Toronto Device: Libra H2O, Libra Colour	It might be handy to have an "audit" program; one that would compare what was in the database to what was in the file system. I know I saw some weirdness on my wife's Kobo; attaching it to a PC gave a few error messages about seeking past EOF in an ePub, plus, ADE shower errors on some books. I'd drop them through ADE, eject and reattach, and would get the same errors, Finally looked at the drive by hand and saw some 0 length ePubs and other bits like that. Dropped them by hand, cleaned up the images directory, and now all seems fine.

06-21-2010, 08:11 PM	#4
corona Addict Posts: 324 Karma: 25168 Join Date: May 2010 Device: kobo	I have noticed this too, without taking the time to really figure it out. Let's hope the firmware folks are thinking about this. I'm totally talking through my hat here, but it strikes me as a relatively uncomplicated problem.