[FileType Plugin] GetFileName

[Terisa de morgan]

Originally Posted by AnselmD

The file date is "new" and mandatory?

New but not mandatory. And this zip file doesn't have the bug.

[AnselmD]

Originally Posted by Terisa de morgan

New but not mandatory. And this zip file doesn't have the bug.

I changed Miscellaneous -> "Max. simultaneous conversion/new Download/plugin jobs" to 1.




The next file is a 7zip file, 7z as extension is not supported as attachment, so I add a .zip extension. Uncompressed it is 42,005,756 bytes.
debuglog.7z.zip

[BetterRed]

For diagnostic purposes only, I'd like to suggest that AnselmD repeats his test with the Include folder option checked.

BR

[Terisa de morgan]

No need, @BetterRed. I think I know which the problem can be, but I don't know an easy way to correct it (different threads with different instances of the auxiliary table). A need the auxiliary table to get the original folder. The best I can do (because I have a limited control over the way I can send information between methods) is omiting some checks if folder is not desired, which is @AnselmD's option.

[Giuseppe Chillem]

Hi Terisa, if I could be of any help, I am facing the same problems.
Random group of books have not filename and path information at all.

Just download 100 random books from all around the internet, add them and the problem will popup (Windows 8.1 32 bit here)

[AnselmD]

Originally Posted by BetterRed

For diagnostic purposes only, I'd like to suggest that AnselmD repeats his test with the Include folder option checked.

BR

@BetterRed, I forgot to start in debug mode. Do you like to have the debug log?

[BetterRed]

Originally Posted by AnselmD

@BetterRed, I forgot to start in debug mode. Do you like to have the debug log?

No thanks - I wouldn't understand it anyway. But it appears that Terisa de M knows what's happening

Originally Posted by Terisa de morgan

No need, @BetterRed. I think I know which the problem can be, but I don't know an easy way to correct it (different threads with different instances of the auxiliary table). A need the auxiliary table to get the original folder. The best I can do (because I have a limited control over the way I can send information between methods) is omiting some checks if folder is not desired, which is @AnselmD's option.

I was wondering if it was a thread race condition. But then I thought - nah, it can't be that because adding books happens in the main thread.

Thread racing came up in another context that might have been related to add books or inter-library copying. I'll do some searches.

Added : Nuh, they were related to calibre startup - I searched for 'race' in KG's posts - ignoring his recent references to Amazon Arms Race.

It appears to be another problem for the master magister.

BR

[BetterRed]

FWIW: I'm not seeing this problem, I add books in relatively small batches (10-30) from multiple directories (my file manager has a browse flat feature) via a Windows shortcut to calibre.exe in Windows Send To - which effectively means I add via

calibre.exe '"book1path.xxx", '"book2path.xxx", '"book2path.xxx", '"book3path.xxx", . . ., '"bookNNpath.xxx".

I also have the Folder Path checked. Not sure if either are relevant.

BR

[jhowell]

In case the problem isn't fixed yet, one cause of random plugin errors can be producing print output when not in debug mode.

https://www.mobileread.com/forums/sh...d.php?t=281382

[kovidgoyal]

Adding books actually happens multiple stages. First ina separate thread, the filesystem is scanned for files. Then in a separate worker process for reading metadata then once that is done the books are added to the database in the GUI thread. @Terisa: what exactly is the problem? Sorry, I'm a little too busy at the moment to read through this thread.

[AnselmD]

I would like to recall my first message into memory:

Originally Posted by AnselmD

Some weeks ago, i started to import epubs in a bulk. I recognized, adding books with the options "add from a single directory" does not work always. Some books did not get a filename. Changing the strategy to "automatic adding" and it works perfect.

Since yesterdays i am not able to use this workaround. Pushing 3700 files to the automatic directory, leads to an
[Errno 24] Too many open files

see:
Bug #1664978 “[Errno 24] Too many open files / automatic adding...” : Bugs : calibre
https://bugs.launchpad.net/calibre/+bug/1664978

So i tried it again with (Version 0.0.11), using "add from a single directory".
But the result was, 20% of the books (all epub) has no filename.

This is the same result I recognized a few weeks ago, this drops to 0%, using automatic adding - which i can not use any more.

I have to supplement it: with auto adding it only worked, if I put the books to the auto folder, before I start calibre. If I moved a lot of books to the auto folder, while calibre is running, I also got missing filenames.

[Terisa de morgan]

Originally Posted by kovidgoyal

Adding books actually happens multiple stages. First ina separate thread, the filesystem is scanned for files. Then in a separate worker process for reading metadata then once that is done the books are added to the database in the GUI thread. @Terisa: what exactly is the problem? Sorry, I'm a little too busy at the moment to read through this thread.

Sorry Kovid, I hadn't really explained it.

The problem here is passing info form the run method to the postadd method (the original path folder). For doing that, the run method includes a dictionary entry with name/original folder and the postadd method reads the dictionary, search the entry related to the file and get the directory. If it doesn't find the entry it assumes that it has processed it (in previous calibre versions, the postadd method was called more than once for a file). The dictionary is included in the preferences method.

In this log, I see a bunch of run call and, in the middle, some postadd calls. It looks like at some point, the postadd method gets a dictionary without some of the last entries and, when it saves it without the processed file, all the intermediate entries are lost.

The solution I think would be initialize the dictionary when calibre starts and not removing anything in the postadd method. As I don't see now more than one call to postadd method, that wouldn't be a problem. The problem would be to clean the dictionary after finishing the adding (or before the beginning). I don't know where to include the cleaning because my initial idea, init, is not valid as it looks like it is called once for file.

This only happens with big batches, there is no problem with small batches as this race situation does not happen.

If somebody has a better idea for passing information between methods, I would be glad to try it.

[kovidgoyal]

The safest way to pass data between the calls is to use the filesystem. Create a temporary file, use the file locking in calibre.utils.lock.ExclusiveFile to write to the file from the run() method and read from it in the postadd() method.

[Terisa de morgan]

Originally Posted by kovidgoyal

The safest way to pass data between the calls is to use the filesystem. Create a temporary file, use the file locking in calibre.utils.lock.ExclusiveFile to write to the file from the run() method and read from it in the postadd() method.

Thank you very much, I was trying it to avoid for time response, but it's really the safest way: semaphores.

[AnselmD]

Originally Posted by AnselmD

Hi,

Some weeks ago, i started to import epubs in a bulk. I recognized, adding books with the options "add from a single directory" does not work always. Some books did not get a filename. Changing the strategy to "automatic adding" and it works perfect.

Since yesterdays i am not able to use this workaround. Pushing 3700 files to the automatic directory, leads to an
[Errno 24] Too many open files

see:
Bug #1664978 “[Errno 24] Too many open files / automatic adding...” : Bugs : calibre
https://bugs.launchpad.net/calibre/+bug/1664978

I do not get "[Errno 24] Too many open files", if I disable the plugin filegetname (without it, I can add ~4600 books using automatic adding).
I would like to start some test with an old version, starting somewhere before the "file date" was introduced. Is there a place to get some older versions?