Problem with Calibre Library

[At_Libitum]

Since it seems to be going around I assume it might be easier to just tag it to this topic. I sadly have to get in line with the others as I just lost a library too...

That is, the booklist is empty after switching back from another lib. All physical folders and files are still present. Calibre also lists each and every author/title when I do a check library as "extra titles"

Steps I did just prior,
I was moving some books to another library(copy+delete) and received an error about a file not being accessible (windows error 5) so did an "open folder" on the book and found there was nothing the matter with the file that I could see, it was readable, had no specific security setting or encryption. Could move it and copy it myself without objections. So I assume it encountered a temporary deadlock situation where two or more processes/threads were trying to access the same file?

So closed the folder window, switched to the library I was moving to so I could check what Calibre did move. Found the file there too (no duplicate, target library was freshly made before move) and switched back to source lib...empty list...AI!

Now I am trying to get back from a Restore Library, it does find every book but it gives me this...

Traceback (most recent call last):
  File "site-packages\calibre\db\restore.py", line 131, in load_preferences
  File "site-packages\calibre\db\restore.py", line 28, in __init__
  File "site-packages\calibre\db\backend.py", line 321, in __init__
ValueError: Path to library too long. Must be less than 89 characters.

eh...but....but...but... my library path (Y:\Books\Transfer) is only 1,2,3... eh 17 chars long?


and everything below that I did not touch. All Calibre's creation. So anything too long is not my doing. Meanwhile I cannot recover as I do not get a helpful hint which folder name or book title might then be too long. Neither does it offer to correct this and shorten the name itself. Trying to go in there manually and hoping I can find what it considers too long I don't dare as the credo goes "DONT TOUCH" so I won't.

But also ain't too looking forward to go import each and every book by hand...adding them via Add Books is likely going to yield the same error and I got more formats per single book folder...and that format ain't addable in bulk.

[theducks]

Originally Posted by At_Libitum

Since it seems to be going around I assume it might be easier to just tag it to this topic. I sadly have to get in line with the others as I just lost a library too...

That is, the booklist is empty after switching back from another lib. All physical folders and files are still present. Calibre also lists each and every author/title when I do a check library as "extra titles"

Steps I did just prior,
I was moving some books to another library(copy+delete) and received an error about a file not being accessible (windows error 5) so did an "open folder" on the book and found there was nothing the matter with the file that I could see, it was readable, had no specific security setting or encryption. Could move it and copy it myself without objections. So I assume it encountered a temporary deadlock situation where two or more processes/threads were trying to access the same file?

So closed the folder window, switched to the library I was moving to so I could check what Calibre did move. Found the file there too (no duplicate, target library was freshly made before move) and switched back to source lib...empty list...AI!

Now I am trying to get back from a Restore Library, it does find every book but it gives me this...

Traceback (most recent call last):
  File "site-packages\calibre\db\restore.py", line 131, in load_preferences
  File "site-packages\calibre\db\restore.py", line 28, in __init__
  File "site-packages\calibre\db\backend.py", line 321, in __init__
ValueError: Path to library too long. Must be less than 89 characters.

eh...but....but...but... my library path (Y:\Books\Transfer) is only 1,2,3... eh 17 chars long?


and everything below that I did not touch. All Calibre's creation. So anything too long is not my doing. Meanwhile I cannot recover as I do not get a helpful hint which folder name or book title might then be too long. Neither does it offer to correct this and shorten the name itself. Trying to go in there manually and hoping I can find what it considers too long I don't dare as the credo goes "DONT TOUCH" so I won't.

But also ain't too looking forward to go import each and every book by hand...adding them via Add Books is likely going to yield the same error and I got more formats per single book folder...and that format ain't addable in bulk.

If the Books are there AND metadata.opf, cover.jpg
are there:
(assume you are viewing the Empty Library in Calibre: No device connected)

Right click the Library Icon: Library Maintenance: Restore database

Let it run

[At_Libitum]

Thanks for the suggestion, but

Originally Posted by At_Libitum

[....snippped....]

Now I am trying to get back from a Restore Library, it does find every
book but it gives me this...

[....snipped]

And The only thing that I could think of doing was to make a 1character long folder in the root of the drive. Move the content of the library folder in there. Mount it in Calibre and repeat the "Restore Library"...

same result...

calibre, version 1.7.0
ERROR: Failed: Restoring database failed, click Show details to see details

Traceback (most recent call last):
  File "site-packages\calibre\db\restore.py", line 112, in run
  File "site-packages\calibre\db\restore.py", line 228, in create_cc_metadata
  File "site-packages\calibre\db\restore.py", line 28, in __init__
  File "site-packages\calibre\db\backend.py", line 321, in __init__
ValueError: Path to library too long. Must be less than 89 characters.

Sorry, path to the lib folder is 1 char, plus 3 for the drive, colon and root char. any shorter and I have a virtual library.

[theducks]

Originally Posted by At_Libitum

Thanks for the suggestion, but




And The only thing that I could think of doing was to make a 1character long folder in the root of the drive. Move the content of the library folder in there. Mount it in Calibre and repeat the "Restore Library"...

same result...

calibre, version 1.7.0
ERROR: Failed: Restoring database failed, click Show details to see details

Traceback (most recent call last):
  File "site-packages\calibre\db\restore.py", line 112, in run
  File "site-packages\calibre\db\restore.py", line 228, in create_cc_metadata
  File "site-packages\calibre\db\restore.py", line 28, in __init__
  File "site-packages\calibre\db\backend.py", line 321, in __init__
ValueError: Path to library too long. Must be less than 89 characters.

Sorry, path to the lib folder is 1 char, plus 3 for the drive, colon and root char. any shorter and I have a virtual library.

Are you renaming the path or moving the library contents (the folder name is the 'Library' name?
This is a Local DRIVE (and NOT shadowed to a backup service)
This Drive has passed chkdisk?

What was the last book added? verify that there is not a huge name there .
Also verify there are NOT any Links inside the Library folders.

[At_Libitum]

Every file is still in place, it's a local drive, passes chkdsk and several other. No links or shortcuts in the library, no humongously long names (did an expand all on the top level folder.


The Y:\0 folder I created in the root I created by letting Calibre create an empty library, then I copied the contents of the Y:\Books\Transfer folder, so basically all of the subfolders but not the metadata.db and metadata_db_prefs_backup as these are likely corrupted anyways if the booklist is empty. All of the books still have their opf files present in their associated folders.

Just for kicks I created again a 1 character long folder in the root. Copied in just one author folder with 1 book folder (and everything there, .opf, cover, epub)
Andre Kyte - The Hidden Realm

same result. same error about the path having to be < 89

All of the test library folders created in the process where created by creating a blank library copying the structure of an existing one (which passes all of the Calibre checks)

Sorry, I ain't going to waste an evening hunting this down. There's clearly an obscure situation where during a batch move files do not get moved due to an external system error (windows error 5) but the booklist still got emptied as if they were moved. And Restore not working...I'll just lift out all of the books in a search and will dump them on a new empty lib, less aggravation from having to recreate missing metadata as from having to figure why a 1char long libfolder name can possibly be too long, and author/title folders? Well Calibre made these itself so if there's no bug in that part of the code, these could not be too long as this would have violated it's own length limitations.

EDIT: And I also just did a check if there weren't any remnants from exploded epubs left behind from Tweak/Modify. None found so that ain't causing a length violation either

[speakingtohe]

It might be just as fast to create a new library and import the contents. Use the copy structure from current library and the second option in add books.

Helen

[DoctorOhh]

Moderator Notice
Moved this post to its own thread. It is not easier to jump on an existing thread when the two problems are entirely different. One is on OSX one is on Window. One person says his entire file structure is gone another acknowledges their file structure is intact.

Originally Posted by At_Libitum

Since it seems to be going around I assume it might be easier to just tag it to this topic. I sadly have to get in line with the others as I just lost a library too...

I guess we will have to wait and see if it is going around, but history shows us that this most likely is not a calibre bug and certainly your library having all the books and files intact is not missing as the other poster claimed his library was missing.

When a library appears empty it is because the metadata.db file has been moved or is corrupt.

[BetterRed]

Originally Posted by DoctorOhh

Moved this post to its own thread. It is not easier to jump on an existing thread when the two problems are entirely different. One is on OSX one is on Window. One person says his entire file structure is gone another acknowledges their file structure is intact.

Its rarely best practice to jump onto an existing thread - unless its to post a

As often as not, whilst one believes one understands what the thread originator wrote, its often the case that what one reads is not what the thread originator meant

Besides which it draws attention away from the OPs problem, which IMO is bad form.

BR

[At_Libitum]

Sorry about that, like said, sometimes what the OP meant and what was read are two different things.


@Helen,

Originally Posted by speakingtohe

It might be just as fast to create a new library and import the contents. Use the copy structure from current library and the second option in add books.

Helen

Thanks for the suggestion. That might indeed have been an option but for the fact that I have multiple formats of the same book and that option expects only one book per folder where as I have several copies of the same book but different formats in the one book folder. So that won't work. What is missing here is a combination of the second and third option (multiple books per folder where each book is the same book in different format)

As for the main issue. I didn't bother to dig deeper what happened. I've recovered all books by simply doing a search all files and dropping the result on a nnew blank library window. I'll worry bout the metadata.opf's later.

I've got a fairly good idea what happened but since it's an obscure situation and prolly not occurring frequent enough it's likely going to take a while before the cause is found. That its a bug I am pretty confident about because files do not get corrupted just by themselves and present day file systems are reliable enough to rule out system corruption.

As I said I got a "windows error 5" during the copy+delete. Which means there was a deadlock situation (the error means 'access denied') during the copy+delete.
Since I let nothing else but Calibre touch my libraries (having read the various warnings) I am pretty sure that this is something in Calibre or possibly in combination with a plugin.

Now, all that is part 1 of the problem.

Part 2 of the problem is why on earth a 1 character library path can be considered too long when trying to performing a "Recover Library". And that I cannot figure nor even understand. Plus that the error stack trace isn't helping at all in this case as it would have been much more useful if it had reported the title/author it tripped over. Even restarting in debug mode did not gave more information

[DoctorOhh]

Hopefully between you and Kovid some sense can be made of your situation.

[kovidgoyal]

You were trying to restore a library located in a location whose path is longer than specified. That error happens if and only if the path of the library being restored is too long, and it refers to the path to the library not the path to anything inside the library. As can be seen here:

https://github.com/kovidgoyal/calibr...ackend.py#L318

I have changed that error message to include the problem path, so that it is clearer.

And as for your original problem, of a duplicate hardlink causing a spurious file locked message, that was fixed in 1.7, as is documented in the whats new page.

[kovidgoyal]

And I think I figured out why you were getting the restore failure even though your library path was shorter than 89 chars. The restore process creates a temporary library, I'm guessing that the path to the temp folder was longer than 89 chars on your system. IIRC this can happen if your username is very long as the temp folder is typically, some thing like

C:\Users\username\AppData\Local\Temp\calibre_XXXXX X\XXXXXX_library_restore

which is already 75 characters.

I have shortened that path to
C:\Users\username\AppData\Local\Temp\calibre_XXXXX X\XXXXXX_rlib

which saves 11 characters

EDIT: I've further changed the code to first try to create the temp library in the parent folder of the library you are tring to restore, and only if that fails, to use the temporary directory. That should make it much more robust.

[At_Libitum]

Ahh...the temp folder, that option I missed completely and the worst thing is I could have realized this...been seeing the temp folder often enough as of late...

indeed it can be quite long...but not from my username...

reconstructing what then must have happened, the path on my system would have been

C:\Users\Ruud\AppData\Local\Temp\calibrePortableTe mp\calibre_XXXXXX\XXXXXX_library_restore\

which is already over the 89 char limit.

Good to hear the access denied situation also is fixed, just my luck that I had still to apply the 1.7 update and was still running on 1.6

[BetterRed]

@At_Libitum : you can give calibre a dedicated temp folder via the CALIBRE_TEMP_DIR environment variable - see http://manual.calibre-ebook.com/cust...ment-variables

I point it at c:\caltemp\

BR

[At_Libitum]

Originally Posted by BetterRed

@At_Libitum : you can give calibre a dedicated temp folder via the CALIBRE_TEMP_DIR environment variable - see http://manual.calibre-ebook.com/cust...ment-variables

I point it at c:\caltemp\

BR

Great! thanks for the pointer. Had not yet seen that one.