Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 11-02-2011, 04:10 PM   #1
lbutlr
Junior Member
lbutlr began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Feb 2010
Device: iPhone
Library on Case-Sensitive file system on OS X

I've recently moved all my web files onto a case-sensitive file system on OS X and I am noticing that Calibre is having a LOT of issues. I am finding many books that it says don't exist because it is trying to access the path with the wrong case settings.

For example, I was trying to copy "A Phule and his money" to my kindle, but calibre failed. Looking at the folder, it was named "A Phule and His Money (1191)".

Other than going through file by file and folder by folder, I don't see anyway to fix this.

I suppose I can move Calibre to a different drive, but I wanted all my web accessible files to be on a single drive, and for other reasons that has to be a case-sensitive drive.
lbutlr is offline   Reply With Quote
Old 11-02-2011, 05:01 PM   #2
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by lbutlr View Post
I suppose I can move Calibre to a different drive, but I wanted all my web accessible files to be on a single drive, and for other reasons that has to be a case-sensitive drive.
Calibre has no trouble with case-sensitive file systems. It's moving between or sharing case-sensitive and non case-sensitive that's the problem, and that's what you recently did. If you stick to one or the other, you won't have trouble after things get fixed. You can also avoid the problem by never changing only case - always change more than just case.
Starson17 is offline   Reply With Quote
 
Advertisement
Old 11-02-2011, 10:56 PM   #3
lbutlr
Junior Member
lbutlr began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Feb 2010
Device: iPhone
Quote:
Originally Posted by Starson17 View Post
Calibre has no trouble with case-sensitive file systems. It's moving between or sharing case-sensitive and non case-sensitive that's the problem, and that's what you recently did. If you stick to one or the other, you won't have trouble after things get fixed. You can also avoid the problem by never changing only case - always change more than just case.
Yes, I moved from insensitive to sensitive. However, there was only one folderol the source, and only one folder on the destination until Calibre got confused. In many cases Calibre lost the folder information because it was looking for the wrong case in the path. This is not 'no trouble', especially when calibre then went and created a new directory with no files in it.

Calibre has a serial number it embeds in the name of each directory (####), it should look for that if it has trouble finding a folder instead of simply assuming it has the case right.

And there is no way to FIX this other than going through manually and figuring out what case Calibre THINKS the folder and files should have, one by one.

Tedious on over 900 books and no idea how many are 'wrong'.
lbutlr is offline   Reply With Quote
Old 11-02-2011, 11:16 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
On a case insensitive filesystem if you change the case of a title or author a mismatch between the stored ath and the on filesystem path can easily develop (remeber that author folders are shared between multiple books). Then when you moce the library blindly to a case sensitive filesystem...poof.

calibre is developed on linux which has case sensitive filesystems, it most definitely does not have any problems with using either a case sesnitive or a case insensitive filesystem in alone.
kovidgoyal is offline   Reply With Quote
Old 11-03-2011, 03:21 AM   #5
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
I went through this myself when I switched over to a case sensitive file system on OS X as well - the simplest way to fix all the mis-matches is to use the 'Check Library' Library Maintenance feature - it will show you every book that appears to be missing from Calibre's perspective - in general these will be the ones that were subjected to what Kovid just described when your library was on a case in-sensitive system.
ldolse is offline   Reply With Quote
Old 11-03-2011, 11:16 AM   #6
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by lbutlr View Post
Yes, I moved from insensitive to sensitive. ...This is not 'no trouble', especially when calibre then went and created a new directory with no files in it.
You changed the case of some authors or titles on a case-insensitive system. The case insensitive system wrote the first case when calibre asked it to create the folder. Then when you changed the case, calibre asked for the new case to be written and the case-insensitive OS said - "No change here - no need to change the folder name or filenames." There's nothing calibre can do about the refusal of the case insensitive OS to make the change calibre requests.

You would have no trouble if you had stayed on a case-insensitive file system, you would have had no trouble if you had started on a case sensitive system and you could have avoided trouble if you had not made any case-only changes to author/title. I think I was pretty clear - it's moving between them that causes trouble.

Last edited by Starson17; 11-03-2011 at 12:30 PM.
Starson17 is offline   Reply With Quote
Old 11-03-2011, 11:37 AM   #7
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,249
Karma: 6020307
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by Starson17 View Post
You changed the case of some authors or titles on a case-insensitive system. The case insensitive system wrote the first case when calibre asked it to create the folder. Then when you changed the case, calibre asked for the new case to be written and the case-insensitive OS said - "No change here - no need to change the folder name or filenames." There's nothing calibre can do about the refusal of the case insensitive OS to make the change calibre requests.

You would have no trouble if you had stayed on a case-insensitive file system, you would have had no trouble if you had started on a case sensitive system and you could have avoided trouble if you had not made any case-only changes to author/title. I think I was pretty clear - it's moving between them that causes trouble and there's no reasonable way for calibre to avoid that problem.
I think the Library maintenance tools could use a few more (case sensitivity) recovery helper tools when a path appears to be missing.

We know a few Calibre rules:
1) Calibre does not expect to store the same exact author in multiple case variants (because of Windows)
2) Title folders all have Calibre Index numbers.
3) book formats within can only have 1 instance of each format

So why not build a case insensitive directory tree/file list and:
a) see if we can find the expected Index number inside and present it as a Fix Case solution.
b) try and repair the formats filename case.
theducks is offline   Reply With Quote
Old 11-03-2011, 12:03 PM   #8
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by theducks View Post
I think the Library maintenance tools could use a few more (case sensitivity) recovery helper tools when a path appears to be missing.
I haven't tried to recover from this type of error, so I'm not sure how well the current library recovery and/or maintenance tools work. I agree that recovery/maintenance tools can be written to help if the current tools don't do the job.

It may also be possible to at least identify case-only changes dynamically and do a double change internally to prevent this problem. (Write a temp author/name, then rewrite the correct final names to force case-insensitive OS's to store the correct case even after a case-only change.) It would have some performance impact, but I'm not sure how much. I suspect Kovid would accept code to address these issues if anyone is motivated. Perhaps the new backend database code currently in progress will spot ways to deal with this issue.

Last edited by Starson17; 11-03-2011 at 12:27 PM.
Starson17 is offline   Reply With Quote
Old 11-03-2011, 12:19 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by theducks View Post
So why not build a case insensitive directory tree/file list and:
a) see if we can find the expected Index number inside and present it as a Fix Case solution.
b) try and repair the formats filename case.
Why not? Coz' that would be a fair bit of work, and the case of people switching from insensitive to sensitive filesystems after having made lots of case only changes is not common enough to justify the effort to me, but, patches are always welcome
kovidgoyal is offline   Reply With Quote
Old 11-03-2011, 12:24 PM   #10
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,249
Karma: 6020307
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by kovidgoyal View Post
Why not? Coz' that would be a fair bit of work, and the case of people switching from insensitive to sensitive filesystems after having made lots of case only changes is not common enough to justify the effort to me, but, patches are always welcome
I wish I had some (even a tiny bit) Python-foo because this has bit me on a dual boot (you only know you have issues, when NTFS complains about 'duplicate folder' when making a copy of the library. SMB must not be 100% compatible )
theducks is offline   Reply With Quote
Old 11-03-2011, 12:28 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Having some problem that bothers you is the best motivation to learn

Just in case:

Instructions on setting up a development environment: http://manual.calibre-ebook.com/develop.html

The relevant library check code is mostly in library/check_library.py
kovidgoyal is offline   Reply With Quote
Old 11-03-2011, 12:50 PM   #12
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
Having some problem that bothers you is the best motivation to learn
It was my motivation, coupled with encouragement from Kovid and others. I wanted a tiny change - "Booth, John Wilkes" in the filename was being imported as "Wilkes, Booth, John." Then, I wanted to prevent author and title overwrites when fetching online metadata. I was lucky that my solutions involved tiny changes. Small and easily solved problems of that type are ancient history now.

You can't find a more friendly and helpful group than the calibre developers. And despite Kovid's stated limitations on what he's willing to do, it's been my experience that once someone shows a true willingness to tackle an issue and has something that looks like a solution, he's willing to use that code as a framework, fix what needs to be fixed, integrates it as necessary, and may spend more time fixing your solution than it would have taken him to write from scratch - possibly in the hope that future efforts will be more refined and lead to a larger group of developers.
Starson17 is offline   Reply With Quote
Old 11-03-2011, 01:09 PM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
My policy is that if someone has taken the trouble to learn enough of the calibre codebase to submit a patch, the least I can do (provided the patch is not something that cannot be included for other reasons) is put in the work to integrate it. So all that you need to do is check with me whether what you're trying to do is ok first. In this case, as I've already said, it is perfectly ok.
kovidgoyal is offline   Reply With Quote
Old 11-04-2011, 02:20 PM   #14
lbutlr
Junior Member
lbutlr began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Feb 2010
Device: iPhone
Quote:
Originally Posted by ldolse View Post
I went through this myself when I switched over to a case sensitive file system on OS X as well - the simplest way to fix all the mis-matches is to use the 'Check Library' Library Maintenance feature - it will show you every book that appears to be missing from Calibre's perspective - in general these will be the ones that were subjected to what Kovid just described when your library was on a case in-sensitive system.
Ah, thank you for that. That will save me oodles of time.

Or not. The errors it found aren't fixable, so i will have to go through each book and fix it manually.

Suggestion: In library maintenance have a option to 'normalize case for all files'. I'm thinking fixing this will be faster in the shell.

Last edited by lbutlr; 11-04-2011 at 02:43 PM.
lbutlr is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
K3 search case sensitive only? charonme Amazon Kindle 1 01-28-2011 06:15 PM
K2 system file access dwallbaum Amazon Kindle 2 09-08-2010 05:32 PM
tags / series case sensitive sort zelda_pinwheel Calibre 11 05-10-2009 02:38 PM
Accessories Do not order a k2 case without the hinge system kiriakou Amazon Kindle 51 03-25-2009 11:28 PM
how to clean more disk space in root file system to upgrade system chinaet iRex 1 12-18-2006 04:54 PM


All times are GMT -4. The time now is 02:27 PM.


MobileRead.com is a privately owned, operated and funded community.