Windows Explorer v Calibre - same files, different results

[Sondar]

After my last lambasting, I tread very lightly this time.

I have a problem, and I don't know quite where to turn for help. I suspect that it is a Windows Explorer problem, but can't find much about anything similar on the web.

I have only been using Calibre for a short time, but have loaded several thousand books already. Everything appeared OK until I wanted to put a copy of my files onto another PC.

Copy / Paste, load Calibre, point it at the new folder - no problem!

Or so I thought.

Turns out that the copy / paste dropped about 400 books, and by analysis, it looks like the last 400 or so that I added.

On investigation, it turns out that while the "missing" books can be read, modified, converted, added to a device, whatever, there is not a trace of them in Windows Explorer!

The only hint that I have been able to track down is that apparently, Windows Explorer can, under some circumstances, "lose" files with long file names, and then refuses to list anything else added later.

Tried dirprn to list the files - it must use something related to Windows Explorer, because it reported (listed) the same subset of files as were listed by Windows Explorer.

Tried Save To Disk, and it appeared to find most of the missing stuff, but then I had another problem - larege files (up on the 40 meg+ range) don't appear to make it through - they generate a couple of error messages each - but the process did allow me to recover some of my lost ground.

Where to from here? Do you think that either of these issues are reportable (I'm wondering if it might not be prudent for Calibre to actually truncate file names if this file length is a problem - I know that with Windows Explorer Copy/Paste, I sometimes get a complete failure with long file names, and sometimes get a partial failure with others - I believe that it is something to do with the total path length, so calculating the longest "safe" name for a particular path should be achievable, provided the Windows Explorer tool limits are known).

Please to not attack me over this post - I am fairly computer savvy, but obviously not partivularly Windows 7 or Calibre savvy.

Any constructive feedback would be most welcome.

[itimpi]

First time I have heard of Windows Explorer failing to copy files without at least giving an error message!

Did you copy/paste the Calibre Library folder or just folders from within it? I would normally just do a copy paste of the whole library folder. If it is a path length problem then as long as you make sure that the path length is no longer on the target system I would have though that you would be OK.

As to Save to Disk not saving large files I do not think this is a known problem. If you get an error message its contents might be relevant. Also if they are PDF files you might want to consider disabling the PDF Writer plugin as that has been known to cause problems.

[user_none]

Find on of the books you're not seeing in Windows Explorer. Then, in the detail pane (show it by selecting the book icon in the status bar at the bottom right). Click on "Click to open" by the word path. This will open Windows Explorer to the location of the book. I just want to double check you're looking in the correct place. Often time there is a space or a . or something that puts the book into a folder you don't expect.

[theducks]

Windows does NOT treat long filenames the same across Windows versions (underneath, there is the old 8.3 name. open a command prompt: type DIR /X *.* to see the linking to the 8.3 names ).

IMHO do the copy with the Higher Version of Windows (Pull the copy from the older system, rather than Push it to the newer system).

If that can't be done.
Open a command prompt and use the XCOPY command

[kovidgoyal]

calibre already truncates file names so that the total path length is under 255 chars.

[Sondar]

Thanks, all, for responding.

To clarify somewhat, based on your responses:

@itimpi: I was trying to copy/paste the whole calibre directory - not individual files. Because of the warning elsewhere, I specifically made sure that the destination path was shorter than the source path.
Thank you - I'll "bug" the other issue.

@user_none: I'm not quite sore what you are suggesting, but if I read you correctly, the problem includes the fact that I can't see ther files, nor their directories - either Book or Author, in Windows Explorer, so I can't navigate to them.

@theducks: All of this occurred from within a Windows 7 environment. None of these files have been exposed to any other environment while they have been in my posession.
I'll try to use xcopy to copy the lot, and see how many turn up in the target directory.

@kovidgoyal: There is no issue with DRM here for any of these publications. Oops - that was so high, I mistook it for your response . I wonder if perhaps the problem is to do with the whole directory being moved AFTER it has been created, resulting in longer file name paths? Is there any tool that can run through the structure to make sure that there are no suddenly created long file names? And no - because Windows Explorer doesn't even realise that these files exist, I can't copy them to a shorter directory structure. I could try shortening the existing structure temporarily to see if any currently missing files suddenly reappear - that would tend to prove / disprove that I am looking at a file name length problem.

Thanks, all - I'll get back to you if I find out what's happened - in the mean time, if any one else has any clues as to what is going on here, I'd appreciate any help that I can get.

[itimpi]

All I can say is that I have successfully copied my complete Calibre eBook library with about 13K books each typically in 3 or 4 formats and an associated calibre2opds catalog (totalling around 100K files and 32GB) without issues using an Explorer Copy/Paste although it took a very long time to copy.

I now use the (free) Microsoft RichCopy tool to keep my live Calibre library synced with my backup copy as that minimizes the number of files that need copying. I have also in the past used XCOPY with the option to only copy changed files set. The problem with XCOPY was that it would not remove files from the target that were no longer in my source, whereas RichCopy does if you have the Purge option set. Both of these options are MUCH faster than using Windows Explorer to do the copy.

[Manichean]

I read some time ago that the Windows version of Calibre truncates the filenames more than the OSX or Linux versions do. If you copied the library from another OS, the paths may be too long for Windows to handle, which, coupled with the Explorer bug you noted in your first post, would result in the behaviour you've noticed.

[Sondar]

I THINK that I've cracked the problem, which appears to be a couple of problems. Please check out what I've worked out, and tell me whether it's likely or not.

1. Problem 1 - file lengths

• The file, on loading, is adjusted so that it cannot be more than 255 characters - thanks to kovidgoyal for that bit of information.
  There are two circumstances where the full path name can grow, and I don't believe that these are taken into account:
  1. If the file is converted from (say) a .pdf file to an .epub file (see example below) and
  2. If the setup wizard is used to move the library from a location with a particular path "overhead" to one with a greater path "overhead", such as moving from, say, c:\files to c:\my files (the second path has a larger overhead than the first.
  An actual example of the first situation, from my own library, is
  The Lion, the Witch and the Wardrobe_ Re - Claudia Dutcher Tillman.pdf
  The Lion, the Witch and the Wardrobe_ Re - Claudia Dutcher Tillman.epub
  (the full name was "The Lion, the Witch and the Wardrobe: Reproducible Activity Book"
  You can see that the original, which was the pdb file, was truncated, but when it was converted to a .epub, the title wasn't further truncated (which it should have been).
  The second situation, I can't vouch for, but I can say that when I moved the library from c:.../Calibre to c:/Sony-Calibre a number of files effectively disappeared from Windows Explorer, and when I moved them back to c:\cs, they appear to have become visible again. I think there is a further complication with the directory structure, I'm coming to the belief that the 255 character limit imposed may be on the true directory, but doesn't take into account the "artificial" directory structure that is imposed if the library is dropped into one of the Windows library entries, where the full path can be greater than the path to where the file actually resides (I've found with some marginal sub-directories that I can see them in WE when I look at them from the drive ("d:\" in my case), but don't appear when I look at them from the Library in WE. This last is in effect a third case, but is tied up with the other two cases, and possibly outside of the control of the Calibre software, but is certainly a trap for young players that should be highlighted.
  

1. Problem 2 - Deleting books

And as I have to go out to see my podiatrist now, I'll get back to this later today.

Please consider

[kovidgoyal]

The maximum path length of files created by calibre on windows is 165 characters + the number of characters in the file extension and the number of characters in the numerical id (which we can upper bound by 11=5+6 reasonably). In other words the bounding of file lengths assumes file extensions of a max of 6 characters. This limit is imposed by restricting the author length to < 40 and the title length to < 40. 165=40 + 40 + 40 + 40 + 5 (the 5 is for path separators and a space).

When creating a new library calibre will refuse if you try to create a library whose path is longer than 89 characters.

So that gives 165+89+11 = 265 characters. This is actually 5 characters over the limit, so I can change the code for that.

However, the examples you posted are well short of the limits.

'The Lion, the Witch and the Wardrobe_ Re - Claudia Dutcher Tillman.epub' is only 71 characters. Add another 90 for the directory components and you still have only 161 characters. Which leaves over 90 for the library path. Neither of the library paths you posted comes closes to 90 characters in length.

[Sondar]

Thank you, kovidoyal,
Then my assumptions and reasoning are incorrect? In that case I am completely at a loss as to what is happening. I know that when I use WE to look at the directory, or to copy / paste the library elsewhere, a fairly large number of files do not show themselves, and are missing from the copy that I have made. I can think of no other reason that could cause this problem.

The attachment shows the original library (on the left) and a copy with a much shorter pathname, made by using the Welcome Wizard, from the source on the left. You;ll notice that although similar, they are not the same. WE shows just 320 items at the bottom of the left hand one, while the right hand one (the copy of the left hand library) shows 421 items, and if you look at the bottom of the left hand image, Barack Obama is followed by Beam H Piper, and then Ben Bova, while the right hand image shows Obama followed by Barbara West, Barrie Sosinsky, Belle de Jour and then Ben Bova.

Which brings me to the second issue. I currently have no books in Calibre by Beam H Piper, but I didn't use the Remove Books command to delete his books - I used the Windows Delete key. To me, it looks as though the Delete key will only delete the entry from Calibre, while the Remove Books command will remove them from file, as well. Hence lost authors and lost books.

The copy using Calibre's wizard, by the way, appears to have copied all books that are showing in Calibre, (and left behind those that weren't in Calibre) even if they don't show up in WE, while WE copy / paste copies everything that's there, minus the "hidden" books.

Suggestions please?

[Sondar]

For completion, here is a pair of lists, both using Karen's dirprint, of the Sony-Calibre library, and the cs library, spawned from it, but generated using the Calibre welcome wizard, pointing the wizard at the new directory. The result still has the correct number of books, although there are large discrepancies in the file structures, as you can see.

[chaley]

@sondar: are the security settings identical for all the folders leading to calibre's library? Were any folders in calibre's library created under a different user ID?

What does the check library command (library/library maintenance/check library) have to say about the library with more folders? Does calibre consider them as invalid?

[Sondar]

@chaley: The security settings are identical across all my drives. I only use one user ID on my PC.
Check Library tells me that there are no errors in the Calibre data base. there are 12 errors, see below:

Missing book formats,A Connecticut Yankee in King Arthur's Co (5487),Samuel Longhorn Clemens\A Connecticut Yankee in King Arthur's Co (5487)\A Connecticut Yankee in King Arthur's Co - Samuel Longhorn Clemens.epub

Extra book formats,The Lion, the Witch and the Wardrobe_ Re (4134),Claudia Dutcher Tillman\The Lion, the Witch and the Wardrobe_ Re (4134)\The Lion, the Witch and the Wardrobe_ - Claudia Dutcher Tillman.pdb

Extra book formats,The Lion, the Witch and the Wardrobe_ Re (4134),Claudia Dutcher Tillman\The Lion, the Witch and the Wardrobe_ Re (4134)\The Lion, the Witch and the Wardrobe_ - Claudia Dutcher Tillman.epub

Extra book formats,The Lion, the Witch and the Wardrobe_ Re (4134),Claudia Dutcher Tillman\The Lion, the Witch and the Wardrobe_ Re (4134)\The Lion, the Witch and the Wardrobe_ - Claudia Dutcher Tillman.pdf

Extra book formats,Plague From Space (1229),Harry Harrison\Plague From Space (1229)\Copy (1)Plague From Space - Harry Harrison.rtf

Missing covers files,A Connecticut Yankee in King Arthur's Co (5487),Samuel Longhorn Clemens\A Connecticut Yankee in King Arthur's Co (5487)\A Connecticut Yankee in King Arthur's Co (5487)\cover.jpg

Cover files not in database,Dragondrums (2187),Anne McCaffrey\Dragondrums (2187)\Dragondrums (2187)\cover.jpg

Cover files not in database,Dragonquest (2189),Anne McCaffrey\Dragonquest (2189)\Dragonquest (2189)\cover.jpg

Cover files not in database,Dragonsinger (2193),Anne McCaffrey\Dragonsinger (2193)\Dragonsinger (2193)\cover.jpg

Cover files not in database,Dragonsong (2196),Anne McCaffrey\Dragonsong (2196)\Dragonsong (2196)\cover.jpg

Cover files not in database,The Great Train Robbery (2201),Michael Crichton\The Great Train Robbery (2201)\The Great Train Robbery (2201)\cover.jpg

Cover files not in database,Red Planet (2198),Robert A. Heinlein\Red Planet (2198)\Red Planet (2198)\cover.jpg

As both versions of the database are showing the same list of books - it's only Windows Explorer that appears to have lost the plot, because it doesn't list all the authors and books that the Library does (by an estimated 450 books), I wouldn't expect too much drama in the library maintenance area. I suspect, but am open to contradiction, that the "extra format" is where I've used the delete key to delete an entry, instead of using the Remove Books" command

[transmitthis]

I think "user_none" was suggesting you navigate in Calibri to one of the books that is un find able in Explorer.exe. Then rclick and select "open containing folder"
Then you should see it in Explorer.exe along with the path etc, you may then be able to work out what happened.

But forgive me if thats of no help, as to be honest I'm a bit confused with which book is hidden or not

I always copy and check the files too, then if theres problems I use a verification app to redo the copy. If there are any issues I grab my backup. failing that I get my other back up from dropbox. If all else fails I pop in my clone drive... lol

Hope you can sort it out.