Auto-add weirdness

[Phssthpok]

I have an auto-add directory set up for Calibre, and sometimes when I drop files in it they get added, and sometimes not. Previously Kovid has said that this is likely due to files being locked by e.g. anti-virus scanners. However, I just dropped two files in there; one got added, and one didn't. When I opened the directory to look, I found 4 files there, some of which had been there for nearly two weeks. Now, during those two weeks I have rebooted the machine at least three times (Windows update, anti-virus update, and once when Windows went on strike)and restarted Calibre even more often. A quick look with Process Explorer didn't show any processes with handles open for those files.

Can anyone tell me what might be going on here? Why would Calibre be ignoring some files but not others, even across multiple reboots?

[theducks]

Not a file type that Calibre recognizes as a auto-add book? (default Auto-add setting) Preferences:Auto-add (tab)

[Phssthpok]

Originally Posted by theducks

Not a file type that Calibre recognizes as a auto-add book? (default Auto-add setting) Preferences:Auto-add (tab)

Nope, they're all epubs. Both the added ones and the ignored ones.

[kovidgoyal]

Check the permissions on the files. They need to have both read and write permission.

[Phssthpok]

Originally Posted by kovidgoyal

Check the permissions on the files. They need to have both read and write permission.

Ah, that's it. I add books from archives by opening the archive, selecting the book(s) I want, and extracting to the auto-add directory. I just re-extracted one of the files that didn't add and hey presto, it's marked read-only. This doesn't cause any complaints about permissions when I manually drag-and-drop from the auto-add directory to Calibre and then delete, so I overlooked that possibility...

Many thanks, as usual!

[Phssthpok]

I did some more experimentation... I couldn't find an unzipper that will let me ignore the stored attributes of the files, so I wrote a batch file to reset the attributes. My idea was to set this up to run at regular intervals to prevent things getting stuck for weeks. However, it had no effect. I changed it to touch the file after resetting the attributes -- still no effect. When I add another file to the auto-add directory, both files disappear after a few seconds.

So here's a plea for a new feature: a preferences option to say "I want Calibre to ignore (reset) attributes on book files in the auto-add directory". I only use the auto-add directory for auto-adding (which is unsurprising) and so if I drop books there it means I want them to be added, with no ifs or buts. How about it?

[kovidgoyal]

I'm afraid not. There is no way to make "resetting file attributes" robust across OSes and filesystems. It's up to you to make sure file attributes for files you put there are correct.

Simply have your batch file create and delete a temporary file in the auto-add folder, that should be enough to trigger a rescan. Or do a file rename to a new filename, though whether that is sufficient or not will depend on the file watching semantics of Qt on your platform.

[BetterRed]

@Phssthpok - You're using Windows right

Extract the files to FolderX, run a batch file or a script against FolderX contents that forces RO off (attrib command) and moves the files from FolderX to the Autoadd folder (move command) .

Maybe you could do the extract, attrib and move in one fell swoop with something like a powershell, ahk, or a ptfb script.

I'm not aware of any Windows unarchivers that provide an option to set/unset attributes - apart from the Archive attribute. Although I have a memory cell suggesting that Total Commander might have something - but that would be tantamount to using a jackhammer to crack a peanut.

BR

[Phssthpok]

Originally Posted by kovidgoyal

I'm afraid not. There is no way to make "resetting file attributes" robust across OSes and filesystems. It's up to you to make sure file attributes for files you put there are correct.

Is there some reason why os.chmod() won't do the trick? According to the docs "Although Windows supports chmod(), you can only set the file’s read-only flag with it" which would be sufficient in this case... and of course it works with all Unices. That maybe leaves a handful of diehard OS/2 or VMS users out in the cold, but hey...

Simply have your batch file create and delete a temporary file in the auto-add folder, that should be enough to trigger a rescan. Or do a file rename to a new filename, though whether that is sufficient or not will depend on the file watching semantics of Qt on your platform.

OK, this works. For the benefit of other Windows victims out there, I now have this:

@echo off
cd "\path\to\Calibre\autoadd"
if not exist *.* exit/b
attrib -r -h -s *.*
echo >foo.xxx
del foo.xxx

Then I run this every few minutes with the Task Scheduler.

I'd still prefer a Preferences option though!

[kovidgoyal]

os.chmod will only work if you have permission to alter the file's attributes, which on unix depends on the file's owner/group compared to yours and on windows on the phase of the moon. And I wont even start down the rabbit hole of filesystem specific attributes, read the man page for chattr if you like.

[Phssthpok]

Originally Posted by kovidgoyal

os.chmod will only work if you have permission to alter the file's attributes, which on unix depends on the file's owner/group compared to yours and on windows on the phase of the moon. And I wont even start down the rabbit hole of filesystem specific attributes, read the man page for chattr if you like.

But os.chmod() could surely be used as a "best attempt" to make the files readable and writable -- in this case by clearing the Windows read-only attribute. (And I agree, if any filesystem-specific attributes were involved, I would shut my mouth now.)

If using os.chmod() to grant RW permissions failed for a particular setup, no-one would be any worse off, but in many/most cases where:

a) I have RW access to the auto-add directory so I can add files and see if they are still there later,
b) I'm running Calibre from an account which has RW access to the auto-add directory so Calibre can import the files and delete them,
c) I put files in the auto-add directory because I want the files to be imported by Calibre if at all possible, and
d) only the common read/write attributes need setting for this to happen;

... it should work, and then I would be much better off... wouldn't I?

[kovidgoyal]

No that is an example of magic behavior. It makes things hard to predict. Sure you individually would be better off. But the feature as a whole becomes less predictable. Right now it is simple, incorrect permissions => no auto add.

With your proposal incorrect permissions => may or may not auto-add depending on the phase of the moon.

[Phssthpok]

Originally Posted by kovidgoyal

No that is an example of magic behavior. It makes things hard to predict. Sure you individually would be better off. But the feature as a whole becomes less predictable. Right now it is simple, incorrect permissions => no auto add.

With your proposal incorrect permissions => may or may not auto-add depending on the phase of the moon.

Hmm. Looking at the number of threads about auto-add, it seems that it's pretty magical as it is. If the permissions are right, it adds. But sometimes the permissions are wrong, or even if they're correct, the auto-adder stalls because an anti-virus check was in progress, or... and then the files sit there, which is not the intended behaviour. We end up with "correct permissions => may or may not auto-add depending on the phase of the moon."

OK, maybe these things aren't Calibre's fault. Still, I think that this would if anything make it less magical by removing one of the potential problems for a lot of users (certainly all Unixers, most if not all Windows users). It certainly wouldn't make it behave less well than it does now.

But hey, you're the boss, and I suspect we'll have to agree to differ if this doesn't convince you...

[theducks]

Originally Posted by Phssthpok

Hmm. Looking at the number of threads about auto-add, it seems that it's pretty magical as it is. If the permissions are right, it adds. But sometimes the permissions are wrong, or even if they're correct, the auto-adder stalls because an anti-virus check was in progress, or... and then the files sit there, which is not the intended behaviour. We end up with "correct permissions => may or may not auto-add depending on the phase of the moon."

OK, maybe these things aren't Calibre's fault. Still, I think that this would if anything make it less magical by removing one of the potential problems for a lot of users (certainly all Unixers, most if not all Windows users). It certainly wouldn't make it behave less well than it does now.

But hey, you're the boss, and I suspect we'll have to agree to differ if this doesn't convince you...

Direct Copying from a CD will be Read only

Some things, a user just needs to do.
Calibre deals with normal files

[Phssthpok]

Originally Posted by theducks

Direct Copying from a CD will be Read only

Some things, a user just needs to do.
Calibre deals with normal files

I assume you don't mean to try using a CD as the auto-add directory itself!

And copying a file from a CD to your auto-add directory produces a file without write permission (= "with read-only set" on Windows), which os.chmod() would (should?) fix... so I don't get your point, since my proposal would make it easier to use auto-add to import files from a CD...?