Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 02-22-2017, 03:18 PM   #166
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by AnselmD View Post
I did one error free run today , I will do another tomorrow.
I dont have the icon for removing sweat from my forehead I hope kovid's help has removed the problem.
Terisa de morgan is offline   Reply With Quote
Old 02-22-2017, 03:34 PM   #167
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,457
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by Terisa de morgan View Post
I dont have the icon for removing sweat from my forehead I hope kovid's help has removed the problem.
here yer go ==>>
BetterRed is offline   Reply With Quote
Advert
Old 02-22-2017, 03:38 PM   #168
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Thank you
Terisa de morgan is offline   Reply With Quote
Old 02-24-2017, 12:29 PM   #169
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
Quote:
Originally Posted by Terisa de morgan View Post
I dont have the icon for removing sweat from my forehead I hope kovid's help has removed the problem.

It works error free!





Nevertheless i have too much books and a performance problem. Maybe someone can give me a hint?

I spend some time to speed up the process to add a lot of books. I tried to push the books from gutenberg (~50.000) to calibre.

I put the metadata.db to my SSD, the books to my HD (it does not really speed up, if I put the complete library to SSD and I do not have a big SSD.).

It starts up very fast, but each book takes longer. I aborted after ~18,000 books.


Click image for larger version

Name:	Screenshot from 2017-02-24 18-04-02.png
Views:	212
Size:	71.1 KB
ID:	155275

If I abort the job and starts to add books again it is fast.

Does anyone have an idea, where is the bottleneck? Journalling with sqlite?
My next try is to put metadata.db and temporary files to ramdisk.
AnselmD is offline   Reply With Quote
Old 02-24-2017, 02:40 PM   #170
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Sorry, really but performance is not my field of expertise (not a joke). And I'm very happy that it's error free now. I'll update it in the first post tomorrow or the day after tomorrow.
Terisa de morgan is offline   Reply With Quote
Advert
Old 02-25-2017, 03:24 AM   #171
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
Quote:
Originally Posted by Terisa de morgan View Post
Sorry, really but performance is not my field of expertise (not a joke). And I'm very happy that it's error free now. I'll update it in the first post tomorrow or the day after tomorrow.
Terisa,

It was not my intention to motivate you for performance enhancements. I am interested only for technical reasons. For a solution I will divide the job in smaller jobs. If I would use my default calibre environment and i would do it in one big job, I would need weeks.

If you allow me, I would like to put some of my results as "" to this thread?
AnselmD is offline   Reply With Quote
Old 02-25-2017, 03:30 AM   #172
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by AnselmD View Post
I am interested only for technical reasons.
I say that it's not my expertise because, in my daily job, there are performance test and enhancements and I'm aware that they're way over my head.

Quote:
Originally Posted by AnselmD View Post
If you allow me, I would like to put some of my results as "" to this thread?
No problem, I don't think you're the only one to make those batch additions (I use small batchs because a lot of books without cleaning disturb me a lot).
Terisa de morgan is offline   Reply With Quote
Old 02-25-2017, 05:21 AM   #173
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
Some additional results of my tests to add a big number of books (52,000) to calibre.

Moving DB and temporary directory to ramdisk does not show a speed up. Looking to the import after 20,000 books shows, calibre-debug uses 100% CPU (it uses always 100% CPU), calibre-parallel uses ~0%. I would like to know how much time is used within pythons garbage collector.

Click image for larger version

Name:	Screenshot from 2017-02-25 10-50-28.png
Views:	162
Size:	59.4 KB
ID:	155282

Calibre debug output is at ~6MB. ~30,000 books (1836 MB) left in the temporary add_books directory of calibre. Dict.txt (of GetFilename Plugin) does not change any more: 52,966 lines, size: 1.8MB (using a very short path for the input files).

I tried to cut the calibre debug output to 0 byte with cat /dev/null > tmp(debugout).txt, but it grows up to the full size immediately.


To find out in how many jobs I should divide the big job, I make the following chart, it shows how many hours i will use for 52,000 books depending on the the book count of the small jobs:

Click image for larger version

Name:	Screenshot from 2017-02-25 10-51-24.png
Views:	169
Size:	72.5 KB
ID:	155283
AnselmD is offline   Reply With Quote
Old 02-25-2017, 05:37 AM   #174
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
I tried to enter the customize plugin dialogue and I got the following error:

Traceback (most recent call last):
File "site-packages/calibre/gui2/preferences/plugins.py", line 293, in customize_plugin
File "site-packages/calibre/gui2/preferences/plugins.py", line 371, in modify_plugin
File "site-packages/calibre/customize/__init__.py", line 155, in do_user_config
File "calibre_plugins.getfilename.__init__", line 301, in config_widget
File "calibre_plugins.getfilename.config", line 149, in __init__
File "calibre_plugins.getfilename.config", line 115, in get_library_config
AttributeError: 'str' object has no attribute 'prefs'
AnselmD is offline   Reply With Quote
Old 02-25-2017, 07:45 AM   #175
AnselmD
Zealot
AnselmD began at the beginning.
 
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
Quote:
Originally Posted by Terisa de morgan View Post
I say that it's not my expertise because, in my daily job, there are performance test and enhancements and I'm aware that they're way over my head.



No problem, I don't think you're the only one to make those batch additions (I use small batchs because a lot of books without cleaning disturb me a lot).
You should know ( ), without your plugin, the time behaviour looks similar.
AnselmD is offline   Reply With Quote
Old 02-25-2017, 09:41 AM   #176
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,771
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Optimizing one-shot additions of very large numbers of books has never really been something I have looked at. AS adding themis typically a one-time operation and should be done in smaller batches anyway, that way if something goes wrong/crashes you dont have to redo them all.
kovidgoyal is offline   Reply With Quote
Old 02-26-2017, 03:05 AM   #177
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by AnselmD View Post
You should know ( ), without your plugin, the time behaviour looks similar.
, at least it hasn't worsened the matter, and I agree with kovid that it doesn't make sense trying to tweak a lot of thing to improve an operation which is not very usual.

Last edited by Terisa de morgan; 10-15-2017 at 01:30 PM.
Terisa de morgan is offline   Reply With Quote
Old 03-01-2017, 08:32 PM   #178
Rellwood
Library Breeder (She/Her)
Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.Rellwood ought to be getting tired of karma fortunes by now.
 
Rellwood's Avatar
 
Posts: 1,149
Karma: 1900479
Join Date: Apr 2015
Location: Fullerton, California
Device: Kobo Aura HD (1) PW3 (4) PW3 2019 new edition (1)
It seems that changes in the configuration of columns won't take effect until a restart.

I may be wrong, but when I tried to import a mobi, or azw3, it wasn't until I restarted Calibre after assigning a date column that the dates were importing.

Also, what would cause the modified date to be the same as the current date? I noticed this happening on some of the files.
Rellwood is offline   Reply With Quote
Old 03-02-2017, 12:11 AM   #179
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by Rellwood View Post
It seems that changes in the configuration of columns won't take effect until a restart.

I may be wrong, but when I tried to import a mobi, or azw3, it wasn't until I restarted Calibre after assigning a date column that the dates were importing.
I'll have to check but I think I've made changes to that column (the other for sure) and they have been effective in the same session.

Quote:
Originally Posted by Rellwood View Post
Also, what would cause the modified date to be the same as the current date? I noticed this happening on some of the files.
I have no idea, sorry, the plugin only gets the date offered by the system functions.
Terisa de morgan is offline   Reply With Quote
Old 03-02-2017, 12:34 AM   #180
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Checked. Yes, I should remember wrong, it looks like preferences are not being reread in the same session, sorry. I'll try to find the problem.
Terisa de morgan is offline   Reply With Quote
Reply

Tags
calibredb, calibredb metadata import, plugin

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[FileType Plugin] YVES Bible Plugin ClashTheBunny Plugins 27 01-16-2023 01:25 AM
How to sort quickly books by filetype? construsol Calibre 7 05-24-2014 06:20 PM
Calibre associated with every filetype?! sej7278 Calibre 14 08-09-2011 10:36 AM
Zip Filetype Plugin? rsingley Plugins 7 02-11-2011 05:11 PM
How to register a filetype? plisken iRex 2 09-28-2009 07:16 AM


All times are GMT -4. The time now is 08:15 AM.


MobileRead.com is a privately owned, operated and funded community.