Losing Files - Problem with Naming of Directories

[Philosopher]

I have noticed a problem - it has existed over a couple of the recent upgrades - but I have just now begun to try and figure it out.

I am not sure how, why, or when the problem is occurring - just that it has occurred a couple of times.

The problem is simply this: books previously added to the library and then the metadata updated - seem to disappear.

Not the entry - but the physical ebook file itself.

This is what I know: it appears to have something to do with the writing of the file after the metadata changes - and it appears to result from a problem with directory naming.

What I can see is that in the cases that this happens to - there appears to be two directories with the same name - but there seems to be a slight difference.

For example I get two directories for xyz abc TITLE under a particular author (I believe I have also seen the problem with the naming of the author folder (one example has one directory with the author and another with the same but with all the characters capitalized).

In this example one directory is Xyz Abc and the other is XYz Abc. The slight change in the capitalization appears to be at minimum one cause of this.

I am trying to take notes on more instances - but was hoping someone else has encountered this.

Here is an actual example that I am referring to in this particular. An author directory for ZZ Top has two books within it. There are now THREE folders however. The first has no problem. But the second - which appears to have occurred after I updated a cover in the metadata (and in this case the cover file still exists - in both folders - but the ebook file (a pdf in this instance) has disappeared.

One folder is named "ZZ Top - Guitar Anthology"
The other is named "Zz Top - Guitar Anthology"

I believe, but cannot be certain, that the original directory was the first "ZZ". But at some point the second was created.

I suspect that the problem is related to the new directory creation and the new name (just capitalization changed - for an unknown reason) - and that the file name either has or has not changed with it - and at the time the new folder is created the file is attempted to be written to it - but fails - and at the same time is lost in the original directory.

I have had this problem in a few different cases - it is quite frustrating. I will try and see if I can see any other "patterns" as I go through and re-analyze the problem.

I'll also try and see if I can "make" it happen.

But I wonder if anyone else has found this problem yet.

One thing I need to add - although I am not sure if this is where the issue is originating or not - is that some times I edit the library in windows and sometimes in linux. I am not sure if the issue comes when I switch between the two (for example have the original entry made in windows - and it moves the original ebook correctly there - but then make a change in metadata in llinux - and it attempts to write the directory differently [or vice versa with OS]). But I am not sure if this is when this occurs of if it is within the same OS that the problem occurs.

I'm trying to keep an eye out. I'd appreciate any feedback. [And I am sure this is a bug that someone will want to address - I am just not capable, at this time, of being able to do that kind of programming.]

Thanks

[kovidgoyal]

you're having case mismatch problems. windows is cases insensitive and linux is case sensitive. Sharing the same library between a case sensitive and case insensitive file system is going to cause problems.

[Philosopher]

I am beginning to wonder if the issue is with the linux version. I say this because I added some books from that install - and am now in the windows environment.

One book's entry is clearly there. But it doesn't show any available formats. When I open the directory, however, in this case the file is still there (I am not sure if it will disappear - as in the other problem - but it seems to be related).

What is happening in this instance is that the filename for the ebook file is too long. So when it saved the name of the file in Linux and created the directory and moved the file - it named it in a way that Calibre doesn't "see" it in windows - because while Windows has it show up in the directory in explorer - it can't open it because of the file name length.

I am not sure how this is related to the other issue - beyond it appearing to be an issue with the formatting of the files/folders in terms of naming them. And in this instance clearly being an issue that doesn't have a problem internally within linux - but does not work well across platforms.

I imagine that would be something simple to fix - and certainly it is something that should be fixed.

Making sure that the file/folder names/formats works uniformly across operating systems.

[Philosopher]

Thanks for the reply. I understand that is the problem.

It seems however that this is an issue you would want to address - as I would expect you would want the libraries to be set up in any version or any OS so that they are read the same.

Couldn't you set the linux version up so that it forces it to use the same file naming criteria as windows - to avoid the conflict in case across OSes?

[Philosopher]

Alternatively - is there any way to prevent it from "losing" the original file? It seems that it goes to move and rename it - and winds up taking an existing file and erasing it.

Not sure why that happens or what addressing it that way would entail. But it seems in general that the issue of copying, renaming, moving, combining libraries, etc. - would be best if the program prevented loss of a file when it goes to write the file in a new directory or rename it.

[Philosopher]

Just to follow up - trying to trace down the problem - I have noticed that books ADDED or CHANGED in a linux install MAY be saved in a way that will not translate over to another OS (windows at least) - because of the naming system - not only the case issue but also the length of the file name causes problems - it seems that it would be desirable to LIMIT the linux system to using only naming options that would be read in other versions.

I assume that the goal is to produce a library that can be shared or combined - including across platforms. So this seems logical. Especially when you add the factor that the program causes the file to disappear when some of these issues occur - that seems like something you would want to fix so that it cannot happen. And I would assume that the solution lies in forcing all the versions to use the same naming format - and in particular forcing all versions to use at minimum a format that would be recognizable without problems in Windows (I haven't tried yet - but I am assuming that the problem doesn't affect a Mac install in the same way - if the library was previously created/edited in linux and then opened in Mac OS).

[Philosopher]

Sorry for several brief replies - but I am just thinking through all the issues.

It seems also that you would want to fix this so it cannot happen - for I also could imagine the following. Let's say I recognized the linux naming length problem in windows - and corrected the name before having the library opened in windows.

Wouldn't the next time I worked with the library in linux - if I was in any way to alter that file's metadata (or at least part of it that affected the name or location of the directory or file - that the problem would re-occur?

[kovidgoyal]

maintaining the library across filesystems with different case conventions while still maintaining the ability to base folder/file names of metadata with the correct case is not possible. I'm not going to explain why as it would take too long. Still, if you can think of a way to make it possible (I cannot) feel free to submit a patch, I will be more than happy to look at it. The relevant function is set_path in the file database2.py

If you want to share the same library between linux and windows, put it on a windows filesystem and mount it as case insensitive in linux.

[dmapr]

You could try adding a registry entry for HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Contro l\Session Manager\kernel\obcaseinsensitive
(dword:0) and see if it helps. I've only used it in conjunction with Cygwin, so I don't know if it'll actually help in your case.