Content server empty, but still downloaded nonexistent books

Purple Lady · 05-24-2015, 07:26 PM

I screwed something up and my Calibre Library was empty! I have no idea what I did but all is well now. CC didn't appear to handle the empty library correctly. By empty I mean there were entries but no actual book files.

I noticed there was a problem when I connected to the content server downloaded a few books and when I tried to read I got an error saying it wasn't a proper zip file or something like that. These books had only the default pic of a book for a cover in CC. I then tried to connect to the content server on my pc and saw there were no book files. I would think CC should have seen that there were no actual files.

I'll send you debug logs from my tablet and Calibre.

chaley · 05-25-2015, 03:25 AM

I have no idea what happened here.

From CC's point of view the books (news articles?) existed. It asked calibre for metadata and received something that looked like metadata. It asked calibre for the book file and received something legitimate. I don't log the file length so I don't know whether what it received actually had content or was zero-length.

There is nothing unusual in the debug logs.

Unfortunately calibre debug logs do not contain the content server logs so I can't see what it thought, but I suspect it thought everything was fine. It would report an error if the files weren't there. It doesn't care if the files are zero length.

What it looks like is that the file system under the content server reported files as existing but gave a zero-length file when a file was opened. That would account for the missing cover and the missing content. All the rest comes out of metadata.db, which is read into memory when the content server starts. Any file system hickup after that wouldn't affect providing metadata. Why a file system would behave in this way is a mystery. Is there some kind of sync program involved? Could something have locked the files, perhaps a sync or backup program?

BetterRed · 05-25-2015, 06:31 AM

Maybe the disk cycled down, and the server timed out on fetching cover/content. I've had this happen where data has been on a USB drive, and the drive took a long while to spin up after going into power save standby. I had to run Keep Alive on the drive to stop it happening. Might have been an image or music library program - but wasn't calibre server.

Problem went away when I moved everything to WD Caviar Black's in a couple of USB 3.0 disk docks.

BR

Purple Lady · 05-25-2015, 02:16 PM

Before I had this problem I was removing libraries from Calibres library list that no longer existed. I have a copy of my library sent to dropbox everyday and I may have opened that library in Calibre instead of my regular one when I was done, but I don't see how that could have emptied it. I gave the libraries different names but that didn't help because the name in dropbox now is the name I used to use for my regular library.

I think I'll change the name to include the word dropbox so I don't get confused again.

Unfortunately I deleted the epub files from my Tab S, but I have a mobi on my Nexus I downloaded from the content server around the same time. This mobi caused Moon+ to crash and AlReader gave an error when trying to read it. The mobi is 724 bytes and when I opened it as a text file this is what it contains:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
<title>404 Not Found</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}

#traceback {
color: red;
}
</style>
</head>
<body>
<h2>404 Not Found</h2>
<p>book: 7236 does not have format: MOBI</p>
<pre id="traceback"></pre>
<div id="powered_by">
<span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span>
</div>
</body>
</html>

chaley · 05-25-2015, 03:14 PM

Quote:

Originally Posted by Purple Lady

Unfortunately I deleted the epub files from my Tab S, but I have a mobi on my Nexus I downloaded from the content server around the same time. This mobi caused Moon+ to crash and AlReader gave an error when trying to read it. The mobi is 724 bytes and when I opened it as a text file this is what it contains:
... CLIP ...

That is very interesting. It says that the metadata.db that the content server had open (in memory) said that a book has a format (mobi), but when it went to fetch the format the file did not exist. The content server set a header indicating that the error happened. Unfortunately CC believed the content server when it said it had the format and accepted the content (the HTML you posted) as being the book without checking the header.

I could change CC to check the header, but I am nervous about making the change. It would be fiendishly difficult to test, and if I get it wrong it could break normal downloads. I will look at it.

Purple Lady · 05-25-2015, 03:32 PM

Quote:

Originally Posted by chaley

That is very interesting. It says that the metadata.db that the content server had open (in memory) said that a book has a format (mobi), but when it went to fetch the format the file did not exist. The content server set a header indicating that the error happened. Unfortunately CC believed the content server when it said it had the format and accepted the content (the HTML you posted) as being the book without checking the header.

I could change CC to check the header, but I am nervous about making the change. It would be fiendishly difficult to test, and if I get it wrong it could break normal downloads. I will look at it.

I tried to access the content server from a browser on my tablet but when I tapped on the book description nothing happened. That's when I went to the pc an got some kind of a message that the file didn't exist. Maybe you could see how the browser on the pc knows a file is missing and use that in CC?

Edit: I doubt this will happen very often, so it might be more risky to fix than not fix.

chaley · 05-25-2015, 03:55 PM

Quote:

Originally Posted by Purple Lady

I tried to access the content server from a browser on my tablet but when I tapped on the book description nothing happened. That's when I went to the pc an got some kind of a message that the file didn't exist. Maybe you could see how the browser on the pc knows a file is missing and use that in CC?

How to detect the problem is to check the HTTP headers to see if the response is "normal". This part is easy. The hard part is working out what to do if the response isn't normal.

There is a lot of code between the GUI and where the book is downloaded.

Quote:

Edit: I doubt this will happen very often, so it might be more risky to fix than not fix.

I agree that it won't happen often, but it can happen if someone manually deletes formats from the calibre library. It is worth some time to see if I can detect the error and then do something besides falling over dead, and do so with minimal risk to it functioning correctly in the normal case.

05-24-2015, 07:26 PM	#1
Purple Lady Grand Sorcerer Posts: 5,698 Karma: 16542228 Join Date: Feb 2010 Location: Pennsylvania Device: Huawei MediaPad M5, LG V30, Boyue T80S, Nexus 7 LTE, K3 3G, Fire HD8	Content server empty, but still downloaded nonexistent books I screwed something up and my Calibre Library was empty! I have no idea what I did but all is well now. CC didn't appear to handle the empty library correctly. By empty I mean there were entries but no actual book files. I noticed there was a problem when I connected to the content server downloaded a few books and when I tried to read I got an error saying it wasn't a proper zip file or something like that. These books had only the default pic of a book for a cover in CC. I then tried to connect to the content server on my pc and saw there were no book files. I would think CC should have seen that there were no actual files. I'll send you debug logs from my tablet and Calibre.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Read e-books in content server not download	alfaboy23	Library Management	6	02-25-2015 02:34 AM
Can't get books from content server on Droid	357mag	Calibre	6	08-02-2011 01:36 PM
Content Server - Books only?	Starson17	Recipes	4	11-06-2010 07:35 PM
Content Server - Books only in RAR/ZIP?	Starson17	Calibre	0	11-06-2010 03:01 PM
Any way to tag certain books not to be shared on my content server?	Timber	Calibre	1	07-20-2010 04:54 PM

05-25-2015, 03:25 AM	#2
chaley Grand Sorcerer Posts: 11,742 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	I have no idea what happened here. From CC's point of view the books (news articles?) existed. It asked calibre for metadata and received something that looked like metadata. It asked calibre for the book file and received something legitimate. I don't log the file length so I don't know whether what it received actually had content or was zero-length. There is nothing unusual in the debug logs. Unfortunately calibre debug logs do not contain the content server logs so I can't see what it thought, but I suspect it thought everything was fine. It would report an error if the files weren't there. It doesn't care if the files are zero length. What it looks like is that the file system under the content server reported files as existing but gave a zero-length file when a file was opened. That would account for the missing cover and the missing content. All the rest comes out of metadata.db, which is read into memory when the content server starts. Any file system hickup after that wouldn't affect providing metadata. Why a file system would behave in this way is a mystery. Is there some kind of sync program involved? Could something have locked the files, perhaps a sync or backup program?

05-25-2015, 06:31 AM	#3
BetterRed null operator (he/him) Posts: 20,587 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	Maybe the disk cycled down, and the server timed out on fetching cover/content. I've had this happen where data has been on a USB drive, and the drive took a long while to spin up after going into power save standby. I had to run Keep Alive on the drive to stop it happening. Might have been an image or music library program - but wasn't calibre server. Problem went away when I moved everything to WD Caviar Black's in a couple of USB 3.0 disk docks. BR

05-25-2015, 02:16 PM	#4
Purple Lady Grand Sorcerer Posts: 5,698 Karma: 16542228 Join Date: Feb 2010 Location: Pennsylvania Device: Huawei MediaPad M5, LG V30, Boyue T80S, Nexus 7 LTE, K3 3G, Fire HD8	Before I had this problem I was removing libraries from Calibres library list that no longer existed. I have a copy of my library sent to dropbox everyday and I may have opened that library in Calibre instead of my regular one when I was done, but I don't see how that could have emptied it. I gave the libraries different names but that didn't help because the name in dropbox now is the name I used to use for my regular library. I think I'll change the name to include the word dropbox so I don't get confused again. Unfortunately I deleted the epub files from my Tab S, but I have a mobi on my Nexus I downloaded from the content server around the same time. This mobi caused Moon+ to crash and AlReader gave an error when trying to read it. The mobi is 724 bytes and when I opened it as a text file this is what it contains: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta> <title>404 Not Found</title> <style type="text/css"> #powered_by { margin-top: 20px; border-top: 2px solid black; font-style: italic; } #traceback { color: red; } </style> </head> <body> <h2>404 Not Found</h2> <p>book: 7236 does not have format: MOBI</p> <pre id="traceback"></pre> <div id="powered_by"> <span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span> </div> </body> </html>

Advert

Advert