Library on Case-Sensitive file system on OS X

[lbutlr]

I've recently moved all my web files onto a case-sensitive file system on OS X and I am noticing that Calibre is having a LOT of issues. I am finding many books that it says don't exist because it is trying to access the path with the wrong case settings.

For example, I was trying to copy "A Phule and his money" to my kindle, but calibre failed. Looking at the folder, it was named "A Phule and His Money (1191)".

Other than going through file by file and folder by folder, I don't see anyway to fix this.

I suppose I can move Calibre to a different drive, but I wanted all my web accessible files to be on a single drive, and for other reasons that has to be a case-sensitive drive.

[Starson17]

Originally Posted by lbutlr

I suppose I can move Calibre to a different drive, but I wanted all my web accessible files to be on a single drive, and for other reasons that has to be a case-sensitive drive.

Calibre has no trouble with case-sensitive file systems. It's moving between or sharing case-sensitive and non case-sensitive that's the problem, and that's what you recently did. If you stick to one or the other, you won't have trouble after things get fixed. You can also avoid the problem by never changing only case - always change more than just case.

[lbutlr]

Originally Posted by Starson17

Calibre has no trouble with case-sensitive file systems. It's moving between or sharing case-sensitive and non case-sensitive that's the problem, and that's what you recently did. If you stick to one or the other, you won't have trouble after things get fixed. You can also avoid the problem by never changing only case - always change more than just case.

Yes, I moved from insensitive to sensitive. However, there was only one folderol the source, and only one folder on the destination until Calibre got confused. In many cases Calibre lost the folder information because it was looking for the wrong case in the path. This is not 'no trouble', especially when calibre then went and created a new directory with no files in it.

Calibre has a serial number it embeds in the name of each directory (####), it should look for that if it has trouble finding a folder instead of simply assuming it has the case right.

And there is no way to FIX this other than going through manually and figuring out what case Calibre THINKS the folder and files should have, one by one.

Tedious on over 900 books and no idea how many are 'wrong'.

[kovidgoyal]

On a case insensitive filesystem if you change the case of a title or author a mismatch between the stored ath and the on filesystem path can easily develop (remeber that author folders are shared between multiple books). Then when you moce the library blindly to a case sensitive filesystem...poof.

calibre is developed on linux which has case sensitive filesystems, it most definitely does not have any problems with using either a case sesnitive or a case insensitive filesystem in alone.

[ldolse]

I went through this myself when I switched over to a case sensitive file system on OS X as well - the simplest way to fix all the mis-matches is to use the 'Check Library' Library Maintenance feature - it will show you every book that appears to be missing from Calibre's perspective - in general these will be the ones that were subjected to what Kovid just described when your library was on a case in-sensitive system.

[Starson17]

Originally Posted by lbutlr

Yes, I moved from insensitive to sensitive. ...This is not 'no trouble', especially when calibre then went and created a new directory with no files in it.

You changed the case of some authors or titles on a case-insensitive system. The case insensitive system wrote the first case when calibre asked it to create the folder. Then when you changed the case, calibre asked for the new case to be written and the case-insensitive OS said - "No change here - no need to change the folder name or filenames." There's nothing calibre can do about the refusal of the case insensitive OS to make the change calibre requests.

You would have no trouble if you had stayed on a case-insensitive file system, you would have had no trouble if you had started on a case sensitive system and you could have avoided trouble if you had not made any case-only changes to author/title. I think I was pretty clear - it's moving between them that causes trouble.

[theducks]

Originally Posted by Starson17

You changed the case of some authors or titles on a case-insensitive system. The case insensitive system wrote the first case when calibre asked it to create the folder. Then when you changed the case, calibre asked for the new case to be written and the case-insensitive OS said - "No change here - no need to change the folder name or filenames." There's nothing calibre can do about the refusal of the case insensitive OS to make the change calibre requests.

You would have no trouble if you had stayed on a case-insensitive file system, you would have had no trouble if you had started on a case sensitive system and you could have avoided trouble if you had not made any case-only changes to author/title. I think I was pretty clear - it's moving between them that causes trouble and there's no reasonable way for calibre to avoid that problem.

I think the Library maintenance tools could use a few more (case sensitivity) recovery helper tools when a path appears to be missing.

We know a few Calibre rules:
1) Calibre does not expect to store the same exact author in multiple case variants (because of Windows)
2) Title folders all have Calibre Index numbers.
3) book formats within can only have 1 instance of each format

So why not build a case insensitive directory tree/file list and:
a) see if we can find the expected Index number inside and present it as a Fix Case solution.
b) try and repair the formats filename case.

[Starson17]

Originally Posted by theducks

I think the Library maintenance tools could use a few more (case sensitivity) recovery helper tools when a path appears to be missing.

I haven't tried to recover from this type of error, so I'm not sure how well the current library recovery and/or maintenance tools work. I agree that recovery/maintenance tools can be written to help if the current tools don't do the job.

It may also be possible to at least identify case-only changes dynamically and do a double change internally to prevent this problem. (Write a temp author/name, then rewrite the correct final names to force case-insensitive OS's to store the correct case even after a case-only change.) It would have some performance impact, but I'm not sure how much. I suspect Kovid would accept code to address these issues if anyone is motivated. Perhaps the new backend database code currently in progress will spot ways to deal with this issue.

[kovidgoyal]

Originally Posted by theducks

So why not build a case insensitive directory tree/file list and:
a) see if we can find the expected Index number inside and present it as a Fix Case solution.
b) try and repair the formats filename case.

Why not? Coz' that would be a fair bit of work, and the case of people switching from insensitive to sensitive filesystems after having made lots of case only changes is not common enough to justify the effort to me, but, patches are always welcome

[theducks]

Originally Posted by kovidgoyal

Why not? Coz' that would be a fair bit of work, and the case of people switching from insensitive to sensitive filesystems after having made lots of case only changes is not common enough to justify the effort to me, but, patches are always welcome

I wish I had some (even a tiny bit) Python-foo because this has bit me on a dual boot (you only know you have issues, when NTFS complains about 'duplicate folder' when making a copy of the library. SMB must not be 100% compatible )

[kovidgoyal]

Having some problem that bothers you is the best motivation to learn

Just in case:

Instructions on setting up a development environment: http://manual.calibre-ebook.com/develop.html

The relevant library check code is mostly in library/check_library.py

[Starson17]

Originally Posted by kovidgoyal

Having some problem that bothers you is the best motivation to learn

It was my motivation, coupled with encouragement from Kovid and others. I wanted a tiny change - "Booth, John Wilkes" in the filename was being imported as "Wilkes, Booth, John." Then, I wanted to prevent author and title overwrites when fetching online metadata. I was lucky that my solutions involved tiny changes. Small and easily solved problems of that type are ancient history now.

You can't find a more friendly and helpful group than the calibre developers. And despite Kovid's stated limitations on what he's willing to do, it's been my experience that once someone shows a true willingness to tackle an issue and has something that looks like a solution, he's willing to use that code as a framework, fix what needs to be fixed, integrates it as necessary, and may spend more time fixing your solution than it would have taken him to write from scratch - possibly in the hope that future efforts will be more refined and lead to a larger group of developers.

[kovidgoyal]

My policy is that if someone has taken the trouble to learn enough of the calibre codebase to submit a patch, the least I can do (provided the patch is not something that cannot be included for other reasons) is put in the work to integrate it. So all that you need to do is check with me whether what you're trying to do is ok first. In this case, as I've already said, it is perfectly ok.

[lbutlr]

Originally Posted by ldolse

I went through this myself when I switched over to a case sensitive file system on OS X as well - the simplest way to fix all the mis-matches is to use the 'Check Library' Library Maintenance feature - it will show you every book that appears to be missing from Calibre's perspective - in general these will be the ones that were subjected to what Kovid just described when your library was on a case in-sensitive system.

Ah, thank you for that. That will save me oodles of time.

Or not. The errors it found aren't fixable, so i will have to go through each book and fix it manually.

Suggestion: In library maintenance have a option to 'normalize case for all files'. I'm thinking fixing this will be faster in the shell.