Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 02-17-2017, 11:35 AM   #136
Terisa de morgan
Wizard
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 4,053
Karma: 4080089
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Aura, Kobo Aura One, XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by AnselmD View Post
The file date is "new" and mandatory?
New but not mandatory. And this zip file doesn't have the bug.
Attached Files
File Type: zip GetFileName.zip (17.0 KB, 19 views)

Last edited by Terisa de morgan; 02-17-2017 at 11:39 AM.
Terisa de morgan is offline   Reply With Quote
Old 02-17-2017, 01:28 PM   #137
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 104
Karma: 10
Join Date: Oct 2013
Device: none
Quote:
Originally Posted by Terisa de morgan View Post
New but not mandatory. And this zip file doesn't have the bug.
I changed Miscellaneous -> "Max. simultaneous conversion/new Download/plugin jobs" to 1.

Click image for larger version

Name:	Screenshot from 2017-02-17 18-58-06.png
Views:	30
Size:	150.4 KB
ID:	155068
Click image for larger version

Name:	Screenshot from 2017-02-17 18-58-36.png
Views:	38
Size:	178.2 KB
ID:	155069

The next file is a 7zip file, 7z as extension is not supported as attachment, so I add a .zip extension. Uncompressed it is 42,005,756 bytes.
debuglog.7z.zip
AnselmD is offline   Reply With Quote
Old 02-17-2017, 02:53 PM   #138
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 8,250
Karma: 6993173
Join Date: Mar 2012
Location: Sydney Australia
Device: none
For diagnostic purposes only, I'd like to suggest that AnselmD repeats his test with the Include folder option checked.

BR
BetterRed is offline   Reply With Quote
Old 02-17-2017, 05:26 PM   #139
Terisa de morgan
Wizard
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 4,053
Karma: 4080089
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Aura, Kobo Aura One, XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
No need, @BetterRed. I think I know which the problem can be, but I don't know an easy way to correct it (different threads with different instances of the auxiliary table). A need the auxiliary table to get the original folder. The best I can do (because I have a limited control over the way I can send information between methods) is omiting some checks if folder is not desired, which is @AnselmD's option.
Terisa de morgan is offline   Reply With Quote
Old 02-17-2017, 05:40 PM   #140
Giuseppe Chillem
Groupie
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 161
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Hi Terisa, if I could be of any help, I am facing the same problems.
Random group of books have not filename and path information at all.

Just download 100 random books from all around the internet, add them and the problem will popup (Windows 8.1 32 bit here)
Giuseppe Chillem is offline   Reply With Quote
Advert
Old 02-17-2017, 05:45 PM   #141
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 104
Karma: 10
Join Date: Oct 2013
Device: none
Quote:
Originally Posted by BetterRed View Post
For diagnostic purposes only, I'd like to suggest that AnselmD repeats his test with the Include folder option checked.

BR
@BetterRed, I forgot to start in debug mode. Do you like to have the debug log?

Click image for larger version

Name:	Screenshot from 2017-02-17 23-39-53.png
Views:	34
Size:	245.7 KB
ID:	155077
AnselmD is offline   Reply With Quote
Old 02-17-2017, 06:49 PM   #142
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 8,250
Karma: 6993173
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by AnselmD View Post
@BetterRed, I forgot to start in debug mode. Do you like to have the debug log?
No thanks - I wouldn't understand it anyway. But it appears that Terisa de M knows what's happening

Quote:
Originally Posted by Terisa de morgan View Post
No need, @BetterRed. I think I know which the problem can be, but I don't know an easy way to correct it (different threads with different instances of the auxiliary table). A need the auxiliary table to get the original folder. The best I can do (because I have a limited control over the way I can send information between methods) is omiting some checks if folder is not desired, which is @AnselmD's option.
I was wondering if it was a thread race condition. But then I thought - nah, it can't be that because adding books happens in the main thread.

Thread racing came up in another context that might have been related to add books or inter-library copying. I'll do some searches.

Added : Nuh, they were related to calibre startup - I searched for 'race' in KG's posts - ignoring his recent references to Amazon Arms Race.

It appears to be another problem for the master magister.

BR

Last edited by BetterRed; 02-17-2017 at 06:59 PM.
BetterRed is offline   Reply With Quote
Old 02-17-2017, 07:12 PM   #143
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 8,250
Karma: 6993173
Join Date: Mar 2012
Location: Sydney Australia
Device: none
FWIW: I'm not seeing this problem, I add books in relatively small batches (10-30) from multiple directories (my file manager has a browse flat feature) via a Windows shortcut to calibre.exe in Windows Send To - which effectively means I add via

calibre.exe '"book1path.xxx", '"book2path.xxx", '"book2path.xxx", '"book3path.xxx", . . ., '"bookNNpath.xxx".

I also have the Folder Path checked. Not sure if either are relevant.

BR
BetterRed is offline   Reply With Quote
Old 02-17-2017, 07:30 PM   #144
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 1,305
Karma: 12008533
Join Date: Nov 2011
Location: Florida
Device: iPad Air 2, Kindle PW2, Nexus 7, Fire 5th Gen
In case the problem isn't fixed yet, one cause of random plugin errors can be producing print output when not in debug mode.

https://www.mobileread.com/forums/sh...d.php?t=281382
jhowell is offline   Reply With Quote
Old 02-17-2017, 09:52 PM   #145
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,281
Karma: 7723403
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Adding books actually happens multiple stages. First ina separate thread, the filesystem is scanned for files. Then in a separate worker process for reading metadata then once that is done the books are added to the database in the GUI thread. @Terisa: what exactly is the problem? Sorry, I'm a little too busy at the moment to read through this thread.
kovidgoyal is offline   Reply With Quote
Old 02-18-2017, 01:48 AM   #146
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 104
Karma: 10
Join Date: Oct 2013
Device: none
I would like to recall my first message into memory:

Quote:
Originally Posted by AnselmD View Post

Some weeks ago, i started to import epubs in a bulk. I recognized, adding books with the options "add from a single directory" does not work always. Some books did not get a filename. Changing the strategy to "automatic adding" and it works perfect.

Since yesterdays i am not able to use this workaround. Pushing 3700 files to the automatic directory, leads to an
[Errno 24] Too many open files

see:
Bug #1664978 “[Errno 24] Too many open files / automatic adding...” : Bugs : calibre
https://bugs.launchpad.net/calibre/+bug/1664978

So i tried it again with (Version 0.0.11), using "add from a single directory".
But the result was, 20% of the books (all epub) has no filename.

This is the same result I recognized a few weeks ago, this drops to 0%, using automatic adding - which i can not use any more.
I have to supplement it: with auto adding it only worked, if I put the books to the auto folder, before I start calibre. If I moved a lot of books to the auto folder, while calibre is running, I also got missing filenames.
AnselmD is offline   Reply With Quote
Old 02-18-2017, 02:49 AM   #147
Terisa de morgan
Wizard
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 4,053
Karma: 4080089
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Aura, Kobo Aura One, XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by kovidgoyal View Post
Adding books actually happens multiple stages. First ina separate thread, the filesystem is scanned for files. Then in a separate worker process for reading metadata then once that is done the books are added to the database in the GUI thread. @Terisa: what exactly is the problem? Sorry, I'm a little too busy at the moment to read through this thread.
Sorry Kovid, I hadn't really explained it.

The problem here is passing info form the run method to the postadd method (the original path folder). For doing that, the run method includes a dictionary entry with name/original folder and the postadd method reads the dictionary, search the entry related to the file and get the directory. If it doesn't find the entry it assumes that it has processed it (in previous calibre versions, the postadd method was called more than once for a file). The dictionary is included in the preferences method.

In this log, I see a bunch of run call and, in the middle, some postadd calls. It looks like at some point, the postadd method gets a dictionary without some of the las entries and, when it saves it without the processed file, all the intermediate entries are lost.

The solution I think would be initialize the dictionary when calibre starts and not removing anything in the postadd method. As I don't see now more than one call to postadd method, that wouldn't be a problem. The problem would be to clean the dictionary after finishing the adding (or before the beginning). I don't know where to include the cleaning because my initial idea, init, is not valid as it looks like it is called once for file.

This only happens with big batches, there is no problem with small batches as this race situation does not happen.

If somebody has a better idea for passing information between methods, I would be glad to try it.
Terisa de morgan is offline   Reply With Quote
Old 02-18-2017, 04:14 AM   #148
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,281
Karma: 7723403
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The safest way to pass data between the calls is to use the filesystem. Create a temporary file, use the file locking in calibre.utils.lock.ExclusiveFile to write to the file from the run() method and read from it in the postadd() method.
kovidgoyal is offline   Reply With Quote
Old 02-18-2017, 04:38 AM   #149
Terisa de morgan
Wizard
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 4,053
Karma: 4080089
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Aura, Kobo Aura One, XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by kovidgoyal View Post
The safest way to pass data between the calls is to use the filesystem. Create a temporary file, use the file locking in calibre.utils.lock.ExclusiveFile to write to the file from the run() method and read from it in the postadd() method.
Thank you very much, I was trying to avoid for time response, but it's really the safest way: semaphores.
Terisa de morgan is offline   Reply With Quote
Old 02-19-2017, 03:06 AM   #150
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 104
Karma: 10
Join Date: Oct 2013
Device: none
Quote:
Originally Posted by AnselmD View Post
Hi,

Some weeks ago, i started to import epubs in a bulk. I recognized, adding books with the options "add from a single directory" does not work always. Some books did not get a filename. Changing the strategy to "automatic adding" and it works perfect.

Since yesterdays i am not able to use this workaround. Pushing 3700 files to the automatic directory, leads to an
[Errno 24] Too many open files

see:
Bug #1664978 “[Errno 24] Too many open files / automatic adding...” : Bugs : calibre
https://bugs.launchpad.net/calibre/+bug/1664978
I do not get "[Errno 24] Too many open files", if I disable the plugin filegetname (without it, I can add ~4600 books using automatic adding).
I would like to start some test with an old version, starting somewhere before the "file date" was introduced. Is there a place to get some older versions?
AnselmD is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[FileType Plugin] YVES Bible Plugin ClashTheBunny Plugins 23 08-16-2016 06:53 AM
How to sort quickly books by filetype? construsol Calibre 7 05-24-2014 06:20 PM
Calibre associated with every filetype?! sej7278 Calibre 14 08-09-2011 10:36 AM
Zip Filetype Plugin? rsingley Plugins 7 02-11-2011 05:11 PM
How to register a filetype? plisken iRex 2 09-28-2009 07:16 AM


All times are GMT -4. The time now is 01:19 PM.


MobileRead.com is a privately owned, operated and funded community.