Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 06-05-2011, 08:34 AM   #1
Sondar
Member
Sondar began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2011
Location: Australia
Device: Sony PRS-650, Kobo e-Reader N647
Windows Explorer v Calibre - same files, different results

After my last lambasting, I tread very lightly this time.

I have a problem, and I don't know quite where to turn for help. I suspect that it is a Windows Explorer problem, but can't find much about anything similar on the web.

I have only been using Calibre for a short time, but have loaded several thousand books already. Everything appeared OK until I wanted to put a copy of my files onto another PC.

Copy / Paste, load Calibre, point it at the new folder - no problem!

Or so I thought.

Turns out that the copy / paste dropped about 400 books, and by analysis, it looks like the last 400 or so that I added.

On investigation, it turns out that while the "missing" books can be read, modified, converted, added to a device, whatever, there is not a trace of them in Windows Explorer!

The only hint that I have been able to track down is that apparently, Windows Explorer can, under some circumstances, "lose" files with long file names, and then refuses to list anything else added later.

Tried dirprn to list the files - it must use something related to Windows Explorer, because it reported (listed) the same subset of files as were listed by Windows Explorer.

Tried Save To Disk, and it appeared to find most of the missing stuff, but then I had another problem - larege files (up on the 40 meg+ range) don't appear to make it through - they generate a couple of error messages each - but the process did allow me to recover some of my lost ground.

Where to from here? Do you think that either of these issues are reportable (I'm wondering if it might not be prudent for Calibre to actually truncate file names if this file length is a problem - I know that with Windows Explorer Copy/Paste, I sometimes get a complete failure with long file names, and sometimes get a partial failure with others - I believe that it is something to do with the total path length, so calculating the longest "safe" name for a particular path should be achievable, provided the Windows Explorer tool limits are known).

Please to not attack me over this post - I am fairly computer savvy, but obviously not partivularly Windows 7 or Calibre savvy.

Any constructive feedback would be most welcome.
Sondar is offline   Reply With Quote
Old 06-05-2011, 08:55 AM   #2
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 3,950
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
First time I have heard of Windows Explorer failing to copy files without at least giving an error message!

Did you copy/paste the Calibre Library folder or just folders from within it? I would normally just do a copy paste of the whole library folder. If it is a path length problem then as long as you make sure that the path length is no longer on the target system I would have though that you would be OK.

As to Save to Disk not saving large files I do not think this is a known problem. If you get an error message its contents might be relevant. Also if they are PDF files you might want to consider disabling the PDF Writer plugin as that has been known to cause problems.
itimpi is offline   Reply With Quote
Old 06-05-2011, 09:08 AM   #3
user_none
Sigil & calibre developer
user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.
 
user_none's Avatar
 
Posts: 2,427
Karma: 950001
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
Find on of the books you're not seeing in Windows Explorer. Then, in the detail pane (show it by selecting the book icon in the status bar at the bottom right). Click on "Click to open" by the word path. This will open Windows Explorer to the location of the book. I just want to double check you're looking in the correct place. Often time there is a space or a . or something that puts the book into a folder you don't expect.
user_none is offline   Reply With Quote
Old 06-05-2011, 09:57 AM   #4
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 13,620
Karma: 5126946
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Windows does NOT treat long filenames the same across Windows versions (underneath, there is the old 8.3 name. open a command prompt: type DIR /X *.* to see the linking to the 8.3 names ).

IMHO do the copy with the Higher Version of Windows (Pull the copy from the older system, rather than Push it to the newer system).

If that can't be done.
Open a command prompt and use the XCOPY command
theducks is offline   Reply With Quote
Old 06-05-2011, 10:03 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 24,833
Karma: 4369673
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
calibre already truncates file names so that the total path length is under 255 chars.
kovidgoyal is offline   Reply With Quote
Old 06-06-2011, 02:42 AM   #6
Sondar
Member
Sondar began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2011
Location: Australia
Device: Sony PRS-650, Kobo e-Reader N647
Thanks, all, for responding.

To clarify somewhat, based on your responses:

@itimpi: I was trying to copy/paste the whole calibre directory - not individual files. Because of the warning elsewhere, I specifically made sure that the destination path was shorter than the source path.
Thank you - I'll "bug" the other issue.

@user_none: I'm not quite sore what you are suggesting, but if I read you correctly, the problem includes the fact that I can't see ther files, nor their directories - either Book or Author, in Windows Explorer, so I can't navigate to them.

@theducks: All of this occurred from within a Windows 7 environment. None of these files have been exposed to any other environment while they have been in my posession.
I'll try to use xcopy to copy the lot, and see how many turn up in the target directory.

@kovidgoyal: There is no issue with DRM here for any of these publications. Oops - that was so high, I mistook it for your response . I wonder if perhaps the problem is to do with the whole directory being moved AFTER it has been created, resulting in longer file name paths? Is there any tool that can run through the structure to make sure that there are no suddenly created long file names? And no - because Windows Explorer doesn't even realise that these files exist, I can't copy them to a shorter directory structure. I could try shortening the existing structure temporarily to see if any currently missing files suddenly reappear - that would tend to prove / disprove that I am looking at a file name length problem.

Thanks, all - I'll get back to you if I find out what's happened - in the mean time, if any one else has any clues as to what is going on here, I'd appreciate any help that I can get.

Last edited by Sondar; 06-06-2011 at 02:52 AM.
Sondar is offline   Reply With Quote
Old 06-06-2011, 03:31 AM   #7
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 3,950
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
All I can say is that I have successfully copied my complete Calibre eBook library with about 13K books each typically in 3 or 4 formats and an associated calibre2opds catalog (totalling around 100K files and 32GB) without issues using an Explorer Copy/Paste although it took a very long time to copy.

I now use the (free) Microsoft RichCopy tool to keep my live Calibre library synced with my backup copy as that minimizes the number of files that need copying. I have also in the past used XCOPY with the option to only copy changed files set. The problem with XCOPY was that it would not remove files from the target that were no longer in my source, whereas RichCopy does if you have the Purge option set. Both of these options are MUCH faster than using Windows Explorer to do the copy.
itimpi is offline   Reply With Quote
Old 06-07-2011, 02:11 AM   #8
Manichean
Wizard
Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!
 
Manichean's Avatar
 
Posts: 3,130
Karma: 80446
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
I read some time ago that the Windows version of Calibre truncates the filenames more than the OSX or Linux versions do. If you copied the library from another OS, the paths may be too long for Windows to handle, which, coupled with the Explorer bug you noted in your first post, would result in the behaviour you've noticed.
Manichean is offline   Reply With Quote
Old 06-07-2011, 08:23 PM   #9
Sondar
Member
Sondar began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2011
Location: Australia
Device: Sony PRS-650, Kobo e-Reader N647
I THINK that I've cracked the problem, which appears to be a couple of problems. Please check out what I've worked out, and tell me whether it's likely or not.
  1. Problem 1 - file lengths
  • The file, on loading, is adjusted so that it cannot be more than 255 characters - thanks to kovidgoyal for that bit of information.
    There are two circumstances where the full path name can grow, and I don't believe that these are taken into account:
    1. If the file is converted from (say) a .pdf file to an .epub file (see example below) and
    2. If the setup wizard is used to move the library from a location with a particular path "overhead" to one with a greater path "overhead", such as moving from, say, c:\files to c:\my files (the second path has a larger overhead than the first.
    An actual example of the first situation, from my own library, is
    The Lion, the Witch and the Wardrobe_ Re - Claudia Dutcher Tillman.pdf
    The Lion, the Witch and the Wardrobe_ Re - Claudia Dutcher Tillman.epub
    (the full name was "The Lion, the Witch and the Wardrobe: Reproducible Activity Book"
    You can see that the original, which was the pdb file, was truncated, but when it was converted to a .epub, the title wasn't further truncated (which it should have been).
    The second situation, I can't vouch for, but I can say that when I moved the library from c:.../Calibre to c:/Sony-Calibre a number of files effectively disappeared from Windows Explorer, and when I moved them back to c:\cs, they appear to have become visible again. I think there is a further complication with the directory structure, I'm coming to the belief that the 255 character limit imposed may be on the true directory, but doesn't take into account the "artificial" directory structure that is imposed if the library is dropped into one of the Windows library entries, where the full path can be greater than the path to where the file actually resides (I've found with some marginal sub-directories that I can see them in WE when I look at them from the drive ("d:\" in my case), but don't appear when I look at them from the Library in WE. This last is in effect a third case, but is tied up with the other two cases, and possibly outside of the control of the Calibre software, but is certainly a trap for young players that should be highlighted.
  1. Problem 2 - Deleting books
And as I have to go out to see my podiatrist now, I'll get back to this later today.

Please consider

Last edited by Sondar; 06-07-2011 at 08:27 PM.
Sondar is offline   Reply With Quote
Old 06-07-2011, 09:40 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 24,833
Karma: 4369673
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The maximum path length of files created by calibre on windows is 165 characters + the number of characters in the file extension and the number of characters in the numerical id (which we can upper bound by 11=5+6 reasonably). In other words the bounding of file lengths assumes file extensions of a max of 6 characters. This limit is imposed by restricting the author length to < 40 and the title length to < 40. 165=40 + 40 + 40 + 40 + 5 (the 5 is for path separators and a space).

When creating a new library calibre will refuse if you try to create a library whose path is longer than 89 characters.

So that gives 165+89+11 = 265 characters. This is actually 5 characters over the limit, so I can change the code for that.

However, the examples you posted are well short of the limits.

'The Lion, the Witch and the Wardrobe_ Re - Claudia Dutcher Tillman.epub' is only 71 characters. Add another 90 for the directory components and you still have only 161 characters. Which leaves over 90 for the library path. Neither of the library paths you posted comes closes to 90 characters in length.

Last edited by kovidgoyal; 06-08-2011 at 10:15 AM.
kovidgoyal is offline   Reply With Quote
Old 06-08-2011, 01:24 AM   #11
Sondar
Member
Sondar began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2011
Location: Australia
Device: Sony PRS-650, Kobo e-Reader N647
Thank you, kovidoyal,
Then my assumptions and reasoning are incorrect? In that case I am completely at a loss as to what is happening. I know that when I use WE to look at the directory, or to copy / paste the library elsewhere, a fairly large number of files do not show themselves, and are missing from the copy that I have made. I can think of no other reason that could cause this problem.

The attachment shows the original library (on the left) and a copy with a much shorter pathname, made by using the Welcome Wizard, from the source on the left. You;ll notice that although similar, they are not the same. WE shows just 320 items at the bottom of the left hand one, while the right hand one (the copy of the left hand library) shows 421 items, and if you look at the bottom of the left hand image, Barack Obama is followed by Beam H Piper, and then Ben Bova, while the right hand image shows Obama followed by Barbara West, Barrie Sosinsky, Belle de Jour and then Ben Bova.

Which brings me to the second issue. I currently have no books in Calibre by Beam H Piper, but I didn't use the Remove Books command to delete his books - I used the Windows Delete key. To me, it looks as though the Delete key will only delete the entry from Calibre, while the Remove Books command will remove them from file, as well. Hence lost authors and lost books.

The copy using Calibre's wizard, by the way, appears to have copied all books that are showing in Calibre, (and left behind those that weren't in Calibre) even if they don't show up in WE, while WE copy / paste copies everything that's there, minus the "hidden" books.

Suggestions please?
Attached Thumbnails
Click image for larger version

Name:	Untitled-1.jpg
Views:	69
Size:	173.9 KB
ID:	72483  
Sondar is offline   Reply With Quote
Old 06-08-2011, 02:08 AM   #12
Sondar
Member
Sondar began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2011
Location: Australia
Device: Sony PRS-650, Kobo e-Reader N647
For completion, here is a pair of lists, both using Karen's dirprint, of the Sony-Calibre library, and the cs library, spawned from it, but generated using the Calibre welcome wizard, pointing the wizard at the new directory. The result still has the correct number of books, although there are large discrepancies in the file structures, as you can see.

Last edited by Sondar; 06-11-2011 at 07:24 AM.
Sondar is offline   Reply With Quote
Old 06-08-2011, 06:58 AM   #13
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 4,715
Karma: 800144
Join Date: Jan 2010
Location: France
Device: Many android devices
@sondar: are the security settings identical for all the folders leading to calibre's library? Were any folders in calibre's library created under a different user ID?

What does the check library command (library/library maintenance/check library) have to say about the library with more folders? Does calibre consider them as invalid?
chaley is offline   Reply With Quote
Old 06-09-2011, 03:11 AM   #14
Sondar
Member
Sondar began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2011
Location: Australia
Device: Sony PRS-650, Kobo e-Reader N647
@chaley: The security settings are identical across all my drives. I only use one user ID on my PC.
Check Library tells me that there are no errors in the Calibre data base. there are 12 errors, see below:

Missing book formats,A Connecticut Yankee in King Arthur's Co (5487),Samuel Longhorn Clemens\A Connecticut Yankee in King Arthur's Co (5487)\A Connecticut Yankee in King Arthur's Co - Samuel Longhorn Clemens.epub

Extra book formats,The Lion, the Witch and the Wardrobe_ Re (4134),Claudia Dutcher Tillman\The Lion, the Witch and the Wardrobe_ Re (4134)\The Lion, the Witch and the Wardrobe_ - Claudia Dutcher Tillman.pdb

Extra book formats,The Lion, the Witch and the Wardrobe_ Re (4134),Claudia Dutcher Tillman\The Lion, the Witch and the Wardrobe_ Re (4134)\The Lion, the Witch and the Wardrobe_ - Claudia Dutcher Tillman.epub

Extra book formats,The Lion, the Witch and the Wardrobe_ Re (4134),Claudia Dutcher Tillman\The Lion, the Witch and the Wardrobe_ Re (4134)\The Lion, the Witch and the Wardrobe_ - Claudia Dutcher Tillman.pdf

Extra book formats,Plague From Space (1229),Harry Harrison\Plague From Space (1229)\Copy (1)Plague From Space - Harry Harrison.rtf

Missing covers files,A Connecticut Yankee in King Arthur's Co (5487),Samuel Longhorn Clemens\A Connecticut Yankee in King Arthur's Co (5487)\A Connecticut Yankee in King Arthur's Co (5487)\cover.jpg

Cover files not in database,Dragondrums (2187),Anne McCaffrey\Dragondrums (2187)\Dragondrums (2187)\cover.jpg

Cover files not in database,Dragonquest (2189),Anne McCaffrey\Dragonquest (2189)\Dragonquest (2189)\cover.jpg

Cover files not in database,Dragonsinger (2193),Anne McCaffrey\Dragonsinger (2193)\Dragonsinger (2193)\cover.jpg

Cover files not in database,Dragonsong (2196),Anne McCaffrey\Dragonsong (2196)\Dragonsong (2196)\cover.jpg

Cover files not in database,The Great Train Robbery (2201),Michael Crichton\The Great Train Robbery (2201)\The Great Train Robbery (2201)\cover.jpg

Cover files not in database,Red Planet (2198),Robert A. Heinlein\Red Planet (2198)\Red Planet (2198)\cover.jpg

As both versions of the database are showing the same list of books - it's only Windows Explorer that appears to have lost the plot, because it doesn't list all the authors and books that the Library does (by an estimated 450 books), I wouldn't expect too much drama in the library maintenance area. I suspect, but am open to contradiction, that the "extra format" is where I've used the delete key to delete an entry, instead of using the Remove Books" command

Last edited by Sondar; 06-09-2011 at 03:14 AM.
Sondar is offline   Reply With Quote
Old 06-10-2011, 08:50 PM   #15
transmitthis
Addict
transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.
 
transmitthis's Avatar
 
Posts: 283
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
I think "user_none" was suggesting you navigate in Calibri to one of the books that is un find able in Explorer.exe. Then rclick and select "open containing folder"
Then you should see it in Explorer.exe along with the path etc, you may then be able to work out what happened.

But forgive me if thats of no help, as to be honest I'm a bit confused with which book is hidden or not

I always copy and check the files too, then if theres problems I use a verification app to redo the copy. If there are any issues I grab my backup. failing that I get my other back up from dropbox. If all else fails I pop in my clone drive... lol

Hope you can sort it out.
transmitthis is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Thumbnails in Windows Explorer scruffy Reading and Management 4 10-29-2013 07:30 PM
903 Pocket903 - Only shows 1 GB in Windows Explorer? Dave-UK PocketBook 2 03-21-2011 03:39 PM
PRS-300 Drag and Drop via Windows Explorer jtdavies1000 Sony Reader 4 12-20-2009 11:26 AM
Windows explorer failure jbee Calibre 11 03-21-2009 10:48 PM
Internet Explorer in Windows XP SP2 - more secure? Alexander Turcic Lounge 1 08-11-2004 07:22 AM


All times are GMT -4. The time now is 05:56 AM.


MobileRead.com is a privately owned, operated and funded community.