Automatic adding folder sometimes fails to add

[kaito]

Preferences->Adding Books->Automatic Adding

I leave calibre running 24/7.

I have set an "Automatic Adding" folder which works most of the time, but sometimes when I remotely copy an epub to this folder from across the network it never gets added. But as soon as I rename the epub (from across the network) it then gets added OR if I add another .epub file to the folder then both will be added. What would cause this?

There are times when it does import a single file from the folder, but other times when I need to do the rename. The filename doesn't matter since I try simple filenames like "1.epub" (and rename to 12.epub or 2.epub or anything else and then calibre seems to finally notice it and import it)

Does it relate to how long calibre has been idle? It only seems to happen after calibre has been up for a long time in the background. If I restart calibre it immediately imports any files from that folder, and also immediately imports any single files I add to that folder.

[theducks]

The name might need to be different
BUT
I suspect the net copy never released the file (in use) lock

[kovidgoyal]

On startup calibre scans and adds all files in the auto add folder. As for why it fails to add sometimes, it is most likely a file locking issue as theducks theorized. You can try running calibre in debug mode and see if anything useful is reported when such a failure occurs.


Just by the way, if you want to add books to calibre across the network, the best way to do it is run the embedded content server and just use its UI to add/convert books as needed.

[kaito]

Both computers are Windows 7.
The computer from which the epub was copied is completely shut down, and calibre still doesn't import the file. When I reconnect to the network share and rename the .epub it is immediately imported.

Maybe if calibre fails an import (due to the file being locked while it was copying) it won't try to import it again after the lock is removed? (unless the file is renamed, or if another file is added to trigger a re-read of the folder?)
Anyway, I will try using the content server in the future.

[kovidgoyal]

On windows any software can lock any file for any reason by simply opening it. Therefore calibre will only retry a particular file for a limited amount of time, if all those retries fail, it gives up.

[theducks]

Originally Posted by kaito

Both computers are Windows 7.
The computer from which the epub was copied is completely shut down, and calibre still doesn't import the file. When I reconnect to the network share and rename the .epub it is immediately imported.

Maybe if calibre fails an import (due to the file being locked while it was copying) it won't try to import it again after the lock is removed? (unless the file is renamed, or if another file is added to trigger a re-read of the folder?)
Anyway, I will try using the content server in the future.

As Kovid said. Any reason. A/V scan, cloud sync, Indexing( go to the Auto Add folder properties and disable indexing )

[compurandom]

I'm on a linux machine and occasionally files dropped in the automatic import directory are ignored. This especially happens if the file already existed and I moved it into the directory.

My guess is that the file has an old timestamp and calibre ignores it. If I touch those files (giving them a timestamp of now), calibre will pick them up.

Of course, I also have the reverse problem -- sometimes I save files from the web browser into the autoimport directory, and calibre picks them up before the download is done. I've found a semi-reliable work around for that.

[kovidgoyal]

You really shouldn't download directly into that folder, instead download to another folder and move into the auto add folder. On non-windows platforms, there is no reliable way to know when a file is finished being downloaded, so calibre simply waits a few seconds and then processes the file.

And as far as I know, file timestamps are not considered at all when deciding whether to add a file or not.

[kaito]

Originally Posted by theducks

As Kovid said. Any reason. A/V scan, cloud sync, Indexing( go to the Auto Add folder properties and disable indexing )

Ah yes it could be the anti-virus holding the file for a few seconds (or more if the epub or pdf is large)

Is there no way to re-scan the folder so files that are now unlocked can be imported rather than just sitting there? How does other software handle this situation?

It is much easier to drop files into a folder than load up a web browser and add files to the content server.

[theducks]

Originally Posted by kaito

Ah yes it could be the anti-virus holding the file for a few seconds (or more if the epub or pdf is large)

Is there no way to re-scan the folder so files that are now unlocked can be imported rather than just sitting there? How does other software handle this situation?

It is much easier to drop files into a folder than load up a web browser and add files to the content server.

Exclude that folder from the A/V scan
The original folder (AKA Download) should ALWAYS be scanned, but there should not be a need to scan it again, for a simple move/copy

[BetterRed]

Windows 7/10 and calibre 0.8.? to 3.31.

I make occasional use of Autoadd, I've never known it not do it's job.

Re Indexing: Because it's on a drive (D:) where everything is indexed I've always indexed the Autoadd folder. I don't like setting up exclusions without good reason - excluding D:\Calibre\Autoadd would gain me nothing (see below).

I have Indexing set to only run when the system is idle, which is the default setting. I've come across situations where users have overridden the default when the index is first/re built, and they've not set it back when indexing completes, on small systems that can result in sluggish performance.

I'm unsure if active indexing would interfere with calibre's Autoadd feature, my hunch is that it wouldn't. The indexer doesn't appear to lock the file its currently processing, and doesn't seem to care if they're already locked; if I walk away from my system it indexes files I have open in Word, which definitely locks files it has open, n.b: I have an autosave addon for Word that saves every 5 minutes.

I only use a browser to download if the site won't interoperate with my download manager. One of the features of which is: when the download ends they get explicitly AV scanned; if okay, they get moved out of Downloads to target directories based on source and type. Sometimes I set the target folder for a specific download to the calibre AutoAdd folder, in fact that's my main use of Autoadd.

And because I like its Previous Versions feature, I run Windows 10 Backup on my D: drive, again without exclusions.

BR

[compurandom]

Originally Posted by kovidgoyal

You really shouldn't download directly into that folder, instead download to another folder and move into the auto add folder.

Ya, I know, that's the right way. It's also inconvenient. Most of the time, my work around works, and when it doesn't, I just delete the bad file from calibre and redownload.

On non-windows platforms, there is no reliable way to know when a file is finished being downloaded, so calibre simply waits a few seconds and then processes the file.

In linux, I think the inotify interface will let you know when nothing has the file open anymore. Or you could wait a few seconds after the file stops growing? inotify is also a good way to find out things have been added to the directory without polling it constantly.

And as far as I know, file timestamps are not considered at all when deciding whether to add a file or not.

It could be coincidence that files sat in the autoimport directory for 5 minutes and then were imported immediately after I touched them. Or maybe the timestamp is not relevant, but touching them somehow triggers calibre to do a rescan?

[kovidgoyal]

calibre already uses a file notification interface to watch the directory (not individual files as that is pretty expensive). So when you touch a file, calibre will rescan the directory.

But, as I sai dbefore if you want to see why the files are not being added, run calibre in debug mode, IIRC it prints out a fair bit of logging for auto add.

[compurandom]

Originally Posted by kovidgoyal

calibre already uses a file notification interface to watch the directory (not individual files as that is pretty expensive). So when you touch a file, calibre will rescan the directory.

But, as I sai dbefore if you want to see why the files are not being added, run calibre in debug mode, IIRC it prints out a fair bit of logging for auto add.

inotify is suppose to be designed to be cheap even for large numbers of files, but I can see using it here being more bother than perhaps it is worth.

Next time I have a file that isn't autoimporting, I'll see if I can start calibre in debug mode, but I suspect restarting it might cause the files to be imported anyway, so it might take a couple of tries to figure out exactly how to trigger it.

[kovidgoyal]

Yes, restarting always imports files. And calibre uses Qt filesystem watcher class which uses inotify under the hood, when available on linux, but it only watches for directory changes, which is the lowest common denominator for all platforms calibre supports.