02-22-2017, 03:18 PM | #166 |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
|
02-22-2017, 03:34 PM | #167 |
null operator (he/him)
Posts: 20,625
Karma: 26960534
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
|
Advert | |
|
02-22-2017, 03:38 PM | #168 |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
Thank you
|
02-24-2017, 12:29 PM | #169 | |
Zealot
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
|
Quote:
It works error free! Nevertheless i have too much books and a performance problem. Maybe someone can give me a hint? I spend some time to speed up the process to add a lot of books. I tried to push the books from gutenberg (~50.000) to calibre. I put the metadata.db to my SSD, the books to my HD (it does not really speed up, if I put the complete library to SSD and I do not have a big SSD.). It starts up very fast, but each book takes longer. I aborted after ~18,000 books. If I abort the job and starts to add books again it is fast. Does anyone have an idea, where is the bottleneck? Journalling with sqlite? My next try is to put metadata.db and temporary files to ramdisk. |
|
02-24-2017, 02:40 PM | #170 |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
Sorry, really but performance is not my field of expertise (not a joke). And I'm very happy that it's error free now. I'll update it in the first post tomorrow or the day after tomorrow.
|
Advert | |
|
02-25-2017, 03:24 AM | #171 | |
Zealot
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
|
Quote:
It was not my intention to motivate you for performance enhancements. I am interested only for technical reasons. For a solution I will divide the job in smaller jobs. If I would use my default calibre environment and i would do it in one big job, I would need weeks. If you allow me, I would like to put some of my results as "" to this thread? |
|
02-25-2017, 03:30 AM | #172 |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
I say that it's not my expertise because, in my daily job, there are performance test and enhancements and I'm aware that they're way over my head.
No problem, I don't think you're the only one to make those batch additions (I use small batchs because a lot of books without cleaning disturb me a lot). |
02-25-2017, 05:21 AM | #173 |
Zealot
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
|
Some additional results of my tests to add a big number of books (52,000) to calibre.
Moving DB and temporary directory to ramdisk does not show a speed up. Looking to the import after 20,000 books shows, calibre-debug uses 100% CPU (it uses always 100% CPU), calibre-parallel uses ~0%. I would like to know how much time is used within pythons garbage collector. Calibre debug output is at ~6MB. ~30,000 books (1836 MB) left in the temporary add_books directory of calibre. Dict.txt (of GetFilename Plugin) does not change any more: 52,966 lines, size: 1.8MB (using a very short path for the input files). I tried to cut the calibre debug output to 0 byte with cat /dev/null > tmp(debugout).txt, but it grows up to the full size immediately. To find out in how many jobs I should divide the big job, I make the following chart, it shows how many hours i will use for 52,000 books depending on the the book count of the small jobs: |
02-25-2017, 05:37 AM | #174 |
Zealot
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
|
I tried to enter the customize plugin dialogue and I got the following error:
Traceback (most recent call last): File "site-packages/calibre/gui2/preferences/plugins.py", line 293, in customize_plugin File "site-packages/calibre/gui2/preferences/plugins.py", line 371, in modify_plugin File "site-packages/calibre/customize/__init__.py", line 155, in do_user_config File "calibre_plugins.getfilename.__init__", line 301, in config_widget File "calibre_plugins.getfilename.config", line 149, in __init__ File "calibre_plugins.getfilename.config", line 115, in get_library_config AttributeError: 'str' object has no attribute 'prefs' |
02-25-2017, 07:45 AM | #175 | |
Zealot
Posts: 105
Karma: 10
Join Date: Oct 2013
Device: none
|
Quote:
|
|
02-25-2017, 09:41 AM | #176 |
creator of calibre
Posts: 43,930
Karma: 22669822
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Optimizing one-shot additions of very large numbers of books has never really been something I have looked at. AS adding themis typically a one-time operation and should be done in smaller batches anyway, that way if something goes wrong/crashes you dont have to redo them all.
|
02-26-2017, 03:05 AM | #177 |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
, at least it hasn't worsened the matter, and I agree with kovid that it doesn't make sense trying to tweak a lot of thing to improve an operation which is not very usual.
Last edited by Terisa de morgan; 10-15-2017 at 01:30 PM. |
03-01-2017, 08:32 PM | #178 |
Library Breeder (She/Her)
Posts: 1,161
Karma: 1900479
Join Date: Apr 2015
Location: Fullerton, California
Device: Kobo Aura HD (1) PW3 (4) PW3 2019 new edition (1)
|
It seems that changes in the configuration of columns won't take effect until a restart.
I may be wrong, but when I tried to import a mobi, or azw3, it wasn't until I restarted Calibre after assigning a date column that the dates were importing. Also, what would cause the modified date to be the same as the current date? I noticed this happening on some of the files. |
03-02-2017, 12:11 AM | #179 | |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
Quote:
I have no idea, sorry, the plugin only gets the date offered by the system functions. |
|
03-02-2017, 12:34 AM | #180 |
Grand Sorcerer
Posts: 6,258
Karma: 11768331
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
|
Checked. Yes, I should remember wrong, it looks like preferences are not being reread in the same session, sorry. I'll try to find the problem.
|
Tags |
calibredb, calibredb metadata import, plugin |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[FileType Plugin] YVES Bible Plugin | ClashTheBunny | Plugins | 27 | 01-16-2023 01:25 AM |
How to sort quickly books by filetype? | construsol | Calibre | 7 | 05-24-2014 06:20 PM |
Calibre associated with every filetype?! | sej7278 | Calibre | 14 | 08-09-2011 10:36 AM |
Zip Filetype Plugin? | rsingley | Plugins | 7 | 02-11-2011 05:11 PM |
How to register a filetype? | plisken | iRex | 2 | 09-28-2009 07:16 AM |