EPUBs Missing, Entries Remain???

taosaur · 06-10-2012, 03:42 AM

Just the facts: I keep tripping over entries in my library with an "epub" format listed, but the only things in the folder are the cover and metadata file. I learn this fact by getting "No compatible/convertible format" errors on various operations, then checking with "open containing folder."

Question: Why is this happening?

Long version: I noticed this problem the first and only time I tried using Calibre to send files to a device. Several files popped up with "no compatible format found" errors (paraphrase). All the ones I noticed were files I'd personally imported via "add books," so there definitely was an EPUB file to begin with. At the time, six months ago, I was in a time crunch setting up a device for someone else and decided to just live without those books. No plugins were installed at that time.

Today I was adding books (much of my library is not yet in Calibre, nor is it otherwise organized) and just happened to discover some more MIA files by running the Count Pages plugin. Of the 900ish books that don't yet have page counts, 223 are empty folders

I'm running Count Pages on my entire current library (5000ish epubs) to see if any files have gone missing since the last time I took a page count, so I'll report back if that gives me a different number. The only way I can see to correct this issue with the problem files is use the list that Count Pages kicks out, individually track down each original epub on my hard drive, and manually drop a copy into the Calibre folders.

So what do you think? I don't know whether Calibre is mysteriously deleting books it copies, or never copying some books during big import jobs. So far I cannot find one commonality among all the books that are missing, though sometimes several from the same author or series will be missing.

ETA: Count Pages confirms only 223 missing epubs in my whole library, which seems to support the skipped imports theory.

taosaur · 06-10-2012, 06:36 AM

About all I've been able to turn up is that all but a handful of the AWOL files came from my first two big imports. The initial import was a pre-existing Calibre library, so it's possible it was missing some of the files from the get-go. I manually replaced the files in the few titles on my TBR list and deleted the rest. The holes will likely get filled as I continue to work through my files.

theducks · 06-10-2012, 01:48 PM

Quote:

Originally Posted by taosaur

Just the facts: I keep tripping over entries in my library with an "epub" format listed, but the only things in the folder are the cover and metadata file. I learn this fact by getting "No compatible/convertible format" errors on various operations, then checking with "open containing folder."

Question: Why is this happening?

Long version: I noticed this problem the first and only time I tried using Calibre to send files to a device. Several files popped up with "no compatible format found" errors (paraphrase). All the ones I noticed were files I'd personally imported via "add books," so there definitely was an EPUB file to begin with. At the time, six months ago, I was in a time crunch setting up a device for someone else and decided to just live without those books. No plugins were installed at that time.

Today I was adding books (much of my library is not yet in Calibre, nor is it otherwise organized) and just happened to discover some more MIA files by running the Count Pages plugin. Of the 900ish books that don't yet have page counts, 223 are empty folders

I'm running Count Pages on my entire current library (5000ish epubs) to see if any files have gone missing since the last time I took a page count, so I'll report back if that gives me a different number. The only way I can see to correct this issue with the problem files is use the list that Count Pages kicks out, individually track down each original epub on my hard drive, and manually drop a copy into the Calibre folders.

So what do you think? I don't know whether Calibre is mysteriously deleting books it copies, or never copying some books during big import jobs. So far I cannot find one commonality among all the books that are missing, though sometimes several from the same author or series will be missing.

ETA: Count Pages confirms only 223 missing epubs in my whole library, which seems to support the skipped imports theory.

RUN (now!

) the Library Maintenance (right-click the Library Icon)

That will report all discrepancies between what the DB has recorded and what still exists.
Improper/un-planned

shutdowns can leave files waiting to be written, in cache (Write behind cache is one of the first things I disable. All files must get written before the user gets to do more. yes there is a performance hit, but I have my data

)

GeckoFriend · 06-10-2012, 02:24 PM

Quote:

Originally Posted by theducks

RUN (now!

) the Library Maintenance (right-click the Library Icon)

That will report all discrepancies between what the DB has recorded and what still exists.
Improper/un-planned

shutdowns can leave files waiting to be written, in cache (Write behind cache is one of the first things I disable. All files must get written before the user gets to do more. yes there is a performance hit, but I have my data

)

Where do I disable Write behind cache?

taosaur · 06-10-2012, 03:50 PM

Quote:

Originally Posted by theducks

Improper/un-planned

shutdowns can leave files waiting to be written, in cache (Write behind cache is one of the first things I disable. All files must get written before the user gets to do more. yes there is a performance hit, but I have my data

)

Hmmm, sounds like a likely suspect. Still, I'm thinking I'd rather just make a point of leaving Calibre open for a while after big imports rather than take the multitasking hit.

theducks · 06-10-2012, 04:25 PM

Quote:

Originally Posted by GeckoFriend

Where do I disable Write behind cache?

On XP (other windows should be similar) BTW this mainly applies to internal drives. You should always use the 'Quick removal' for any drive that can be unplugged or powered OFF seperate from the computer. Cat help happens.

GeckoFriend · 06-10-2012, 06:44 PM

Quote:

Originally Posted by theducks

On XP (other windows should be similar) BTW this mainly applies to internal drives. You should always use the 'Quick removal' for any drive that can be unplugged or powered OFF seperate from the computer. Cat help happens.

Interesting; that isn't an option on Vista for C: drive.

theducks · 06-10-2012, 07:38 PM

Quote:

Originally Posted by GeckoFriend

Interesting; that isn't an option on Vista for C: drive.

Let me

that for you.

Same options, just a slight difference in getting there. This is a HW setting option.

PeterT · 06-10-2012, 09:18 PM

I've never ever encountered issues relating to data not being written to a drive; I really think that suggesting disabling of write-through caching on internal hard drives is masking other issues and not solving anything.

theducks · 06-10-2012, 09:40 PM

Quote:

Originally Posted by PeterT

I've never ever encountered issues relating to data not being written to a drive; I really think that suggesting disabling of write-through caching on internal hard drives is masking other issues and not solving anything.

Write Behind Cache is where the Write really has not completed (usually caused by slow devices)
Write Through Cache just leaves a copy in cache (not needed in a normal Save) for faster reads.

06-10-2012, 03:42 AM	#1
taosaur intelligent posterior Posts: 1,562 Karma: 21295618 Join Date: Mar 2009 Location: Ohiopolis Device: Kindle Paperwhite 2, Samsung S8, Lenovo Tab 3 Pro	EPUBs Missing, Entries Remain??? Just the facts: I keep tripping over entries in my library with an "epub" format listed, but the only things in the folder are the cover and metadata file. I learn this fact by getting "No compatible/convertible format" errors on various operations, then checking with "open containing folder." Question: Why is this happening? Long version: I noticed this problem the first and only time I tried using Calibre to send files to a device. Several files popped up with "no compatible format found" errors (paraphrase). All the ones I noticed were files I'd personally imported via "add books," so there definitely was an EPUB file to begin with. At the time, six months ago, I was in a time crunch setting up a device for someone else and decided to just live without those books. No plugins were installed at that time. Today I was adding books (much of my library is not yet in Calibre, nor is it otherwise organized) and just happened to discover some more MIA files by running the Count Pages plugin. Of the 900ish books that don't yet have page counts, 223 are empty folders I'm running Count Pages on my entire current library (5000ish epubs) to see if any files have gone missing since the last time I took a page count, so I'll report back if that gives me a different number. The only way I can see to correct this issue with the problem files is use the list that Count Pages kicks out, individually track down each original epub on my hard drive, and manually drop a copy into the Calibre folders. So what do you think? I don't know whether Calibre is mysteriously deleting books it copies, or never copying some books during big import jobs. So far I cannot find one commonality among all the books that are missing, though sometimes several from the same author or series will be missing. ETA: Count Pages confirms only 223 missing epubs in my whole library, which seems to support the skipped imports theory. Last edited by taosaur; 06-10-2012 at 03:46 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Should I remain a DINOSAUR	Ms M	Kobo Reader	11	07-15-2011 07:03 AM
User Categories remain after deleted	meme	Development	9	02-27-2011 06:39 AM
PRS-505 Missing several lines in converted Epubs	VanPersie	Sony Reader	3	04-25-2010 02:26 PM
Illustrations missing from epubs	alecE	ePub	2	12-28-2009 02:10 PM
Do you think the Internet should remain free?	Colin Dunstan	Lounge	3	09-20-2003 10:31 AM

06-10-2012, 06:36 AM	#2
taosaur intelligent posterior Posts: 1,562 Karma: 21295618 Join Date: Mar 2009 Location: Ohiopolis Device: Kindle Paperwhite 2, Samsung S8, Lenovo Tab 3 Pro	About all I've been able to turn up is that all but a handful of the AWOL files came from my first two big imports. The initial import was a pre-existing Calibre library, so it's possible it was missing some of the files from the get-go. I manually replaced the files in the few titles on my TBR list and deleted the rest. The holes will likely get filled as I continue to work through my files.

06-10-2012, 09:18 PM	#9
PeterT Grand Sorcerer Posts: 12,159 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	I've never ever encountered issues relating to data not being written to a drive; I really think that suggesting disabling of write-through caching on internal hard drives is masking other issues and not solving anything.