Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 03-05-2023, 04:57 PM   #1036
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 11,000
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Thanks. I didn't think of a Calibre version problem. Working now!

Tooltip seems to be misbehaving a bit, though. Running on the same chain as before:

Code:
calibre, version 6.13.0
ERROR: Cannot run chain: Action Chains: conditions for chain (Read) are not met.
tooltip:

Last edited by ownedbycats; 03-05-2023 at 07:59 PM.
ownedbycats is online now   Reply With Quote
Old 03-05-2023, 08:53 PM   #1037
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,196
Karma: 1995558
Join Date: Aug 2015
Device: Kindle
Have you actually specified a value for a tooltip in the condition dialog?
Attached Thumbnails
Click image for larger version

Name:	1.jpg
Views:	832
Size:	115.9 KB
ID:	200139   Click image for larger version

Name:	2.jpg
Views:	825
Size:	17.9 KB
ID:	200140  
capink is offline   Reply With Quote
Old 03-05-2023, 11:07 PM   #1038
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 11,000
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Here's what I see

Click image for larger version

Name:	2023-03-06 00_05_53-Evaluate Condition (only one template dialog allowed).png
Views:	835
Size:	26.1 KB
ID:	200142

Click image for larger version

Name:	2023-03-06 00_06_03-ERROR_ Cannot run chain.png
Views:	858
Size:	8.2 KB
ID:	200143

If relevant, I also have an additional condition on a 'View' action. The chain seems to work as expected when 1 book is selected, though.
Click image for larger version

Name:	2023-03-06 00_08_41-Add actions.png
Views:	838
Size:	24.0 KB
ID:	200145
Click image for larger version

Name:	2023-03-06 00_07_17-Evaluate Condition (only one template dialog allowed).png
Views:	809
Size:	31.3 KB
ID:	200144

Last edited by ownedbycats; 03-05-2023 at 11:11 PM.
ownedbycats is online now   Reply With Quote
Old 03-05-2023, 11:19 PM   #1039
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 11,000
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Oh, tooltip is what I needed to enter. I feel silly now.
ownedbycats is online now   Reply With Quote
Old 03-07-2023, 05:19 PM   #1040
bigwoof
Connoisseur
bigwoof began at the beginning.
 
Posts: 88
Karma: 18
Join Date: Jun 2020
Device: Boox Note Air3, don't use my kindle anymore
How do I run a chain on every book in a large (~58k) library?

Hi,

I created a chain to fix all the various tags in my library. it has 4 different single field edit templates all with default scope (whatever is selected).

I selected all the books and ran the chain. it crashed with an out of memory error.

Is there a way to get the chains to run book by book over an entire library? I don't mind waiting a few/many hours for it to be done.

Thanks!

Bigwoof
bigwoof is offline   Reply With Quote
Old 03-09-2023, 02:22 AM   #1041
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 11,000
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Possible oversight: Copying a blank 'separator' chain creates a chain titled (copy).
ownedbycats is online now   Reply With Quote
Old 03-10-2023, 03:46 PM   #1042
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,196
Karma: 1995558
Join Date: Aug 2015
Device: Kindle
Quote:
Originally Posted by ownedbycats View Post
Possible oversight: Copying a blank 'separator' chain creates a chain titled (copy).
Will be fixed in the next release.
capink is offline   Reply With Quote
Old 03-10-2023, 03:48 PM   #1043
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,196
Karma: 1995558
Join Date: Aug 2015
Device: Kindle
Quote:
Originally Posted by bigwoof View Post
Hi,

I created a chain to fix all the various tags in my library. it has 4 different single field edit templates all with default scope (whatever is selected).

I selected all the books and ran the chain. it crashed with an out of memory error.

Is there a way to get the chains to run book by book over an entire library? I don't mind waiting a few/many hours for it to be done.

Thanks!

Bigwoof
Here is a modified version of the Chain Caller action (called Modified Chain Caller) that can call a target chain multiple times to process chunks of books for each iteration.

Code:
#!/usr/bin/env python
# ~*~ coding: utf-8 ~*~

from qt.core import (QApplication, Qt, QWidget, QGridLayout, QHBoxLayout, QVBoxLayout,
                      QGroupBox, QComboBox, QCheckBox, QSpinBox)

import time

from calibre import prints
from calibre.constants import DEBUG

from calibre_plugins.action_chains.chains import Chain
import calibre_plugins.action_chains.config as cfg


from calibre_plugins.action_chains.actions.chain_caller import ChainCallerAction


tooltip = '''
This option splits the books into multiple chunks and run the target chain multiple times,
each time processing a chunk of books. The chunk size can be selected in the spin box.
The book ids in each chunk is passed to the target chain in a variabe called _book_ids.
You must adjust the scope of the actions in the target chains using a template scope to
act on this variable.
'''

class ConfigWidget(QWidget):
    def __init__(self, plugin_action, chain_name, chains_config, *args, **kwargs):
        QWidget.__init__(self)
        self.plugin_action = plugin_action
        self.gui = plugin_action.gui
        self.db = self.gui.current_db
        self.chain_name = chain_name
        self.chains_config = chains_config
        self._init_controls()

    def _init_controls(self):

        l = self.l = QVBoxLayout()
        self.setLayout(l)

        chain_groupbox = QGroupBox(_('Select chain'))
        l.addWidget(chain_groupbox)
        chain_groupbox_l = QVBoxLayout()
        chain_groupbox.setLayout(chain_groupbox_l)
        self.chain_combo = QComboBox()
        chain_groupbox_l.addWidget(self.chain_combo)

        opts_groupbox = QGroupBox(_('Options'))
        opts_groupbox_l = QVBoxLayout()
        opts_groupbox.setLayout(opts_groupbox_l)
        l.addWidget(opts_groupbox)

        vars_chk = self.vars_chk = QCheckBox(_('Pass variables from calling chain'))
        vars_chk.setChecked(False)
        opts_groupbox_l.addWidget(vars_chk)

        #all_chain_names = [chain_config['menuText'] for chain_config in cfg.get_chains_config()]
        all_chain_names = [chain_config['menuText'] for chain_config in self.chains_config if chain_config['menuText']]
        try:
            all_chain_names.remove(self.chain_name)
        except:
            pass
        self.chain_combo.addItems(all_chain_names)
        self.chain_combo.setCurrentIndex(-1)

        chunk_group_box = self.chunk_group_box = QGroupBox('Split books into chunks')
        chunk_group_box.setCheckable(True)
        chunk_group_box.setChecked(False)
        chunk_group_box_l = QVBoxLayout()
        chunk_group_box.setLayout(chunk_group_box_l)
        l.addWidget(chunk_group_box)
        chunk_group_box.setToolTip(tooltip)
        self.chunk_spinbox = QSpinBox()
        self.chunk_spinbox.setMaximum(100000)
        self.chunk_spinbox.setMinimum(1)
        self.chunk_spinbox.setSingleStep(1)
        self.chunk_spinbox.setValue(10)
        chunk_group_box_l.addWidget(self.chunk_spinbox)

        delay_group_box = self.delay_group_box = QGroupBox('Delay')
        delay_group_box_l = QVBoxLayout()
        delay_group_box.setLayout(delay_group_box_l)
        l.addWidget(delay_group_box)
        delay_group_box.setToolTip(_('Delay in seconds between calling chunks'))
        self.delay_spinbox = QSpinBox()
        self.delay_spinbox.setMaximum(100000)
        self.delay_spinbox.setMinimum(0)
        self.delay_spinbox.setSingleStep(1)
        self.delay_spinbox.setValue(0)
        delay_group_box_l.addWidget(self.delay_spinbox)

        l.addStretch(1)

        self.setMinimumSize(300,300)

    def load_settings(self, settings):
        if settings:
            idx = self.chain_combo.findText(settings['chain_name'])
            self.chain_combo.setCurrentIndex(idx)
            self.vars_chk.setChecked(settings.get('pass_vars', False))
            self.chunk_group_box.setChecked(settings.get('split_to_chunks', False))
            self.chunk_spinbox.setValue(settings.get('chunk_size', 10))
            self.delay_spinbox.setValue(settings.get('delay', 0))

    def save_settings(self):
        settings = {}
        settings['chain_name'] = self.chain_combo.currentText()
        settings['pass_vars'] = self.vars_chk.isChecked()
        settings['split_to_chunks'] = self.chunk_group_box.isChecked()
        settings['chunk_size'] = self.chunk_spinbox.value()
        settings['delay'] = self.delay_spinbox.value()
        return settings

class ModifiedChainCallerAction(ChainCallerAction):

    name = 'Modified Chain Caller'
    is_experimental = False

    def run(self, gui, settings, chain):
        # Chain variables {
        add_to_vars = {}
        if settings.get('pass_vars'):
            add_to_vars = self.sanitize_chain_vars(chain)
        add_to_vars['_caller'] = chain.chain_name
        # }
        book_ids = chain.scope().get_book_ids()
        if settings.get('split_to_chunks', False):
            chunk_size = settings['chunk_size']
            chunks = [book_ids[x:x+chunk_size] for x in range(0, len(book_ids), chunk_size)]
        else:
            chunks = [book_ids]
        chunks_no = len(chunks)
        delay = settings.get('delay', 0)
        for i, chunk in enumerate(chunks):
            prints('Action Chains: Chain Caller: running chunk no {} of {}'.format(i, chunks_no))
            add_to_vars['_book_ids'] = ','.join([str(x) for x in chunk])
            target_chain_config = cfg.get_chain_config(settings['chain_name'])
            target_chain = Chain(self.plugin_action, target_chain_config, show_progress=False,
                                chain_vars=add_to_vars, chain_caller=True)
            target_chain.run()
            del target_chain
            if delay:
                if DEBUG:
                    prints(f'sleeping for {delay} seconds')
                time.sleep(delay)

    def config_widget(self):
        return ConfigWidget
You should use this Modified Chain Caller to call your target chain as follows:
  1. Add the Modified Chain Caller as a custom action using the module editor (Action Chains > manage modules > create module > copy/paste the code above)
  2. Create a new chain that consists of a single action > Modified Chain Caller.
  3. In the settings of the Modified Chain Caller, choose your target chain from the dropdown menu.
  4. In the settings of the Modified Chain Caller, tick the box for splitting books into chunks, and choose the chunk size (default is 10 books)
  5. In your target chain, all the actions must have a template scope with the following template:
    Code:
    program:
        globals(_book_ids)
  6. Now select the books you want to process, and run the Modified Chain Caller chain.

I am not sure whether this can help you with the out of memory problem, give it a try and see.

Last edited by capink; 09-07-2023 at 03:05 PM. Reason: Modify action to option for delay between calling chunks
capink is offline   Reply With Quote
Old 03-10-2023, 07:07 PM   #1044
bigwoof
Connoisseur
bigwoof began at the beginning.
 
Posts: 88
Karma: 18
Join Date: Jun 2020
Device: Boox Note Air3, don't use my kindle anymore
Quote:
Originally Posted by capink View Post
Here is a modified version of the Chain Caller action (called Modified Chain Caller) that can call a target chain multiple times to process chunks of books for each iteration.
Thanks! will try it! I tried some other options and could not quite figure out the right python magic to make it work. interestingly, running my chain on a 1500 books ended up with windows 11 (running on a laptop with 16G ram and 32G ramdisk) rebooting with a rare BSOD...

It works fine with up to 200 books.

Bigwoof
bigwoof is offline   Reply With Quote
Old 03-10-2023, 07:11 PM   #1045
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 11,000
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Might be worth running a memtest...
ownedbycats is online now   Reply With Quote
Old 03-10-2023, 08:46 PM   #1046
bigwoof
Connoisseur
bigwoof began at the beginning.
 
Posts: 88
Karma: 18
Join Date: Jun 2020
Device: Boox Note Air3, don't use my kindle anymore
Quote:
Originally Posted by capink View Post
[*]Create a new chain that consists of a single action > Modified Chain Caller.
I can't seem to get this part to work. Even after restarting calibre multiple times (also threw in a reboot in there as well), I can't see the modules appearing as selectable actions in the chain action list. I created a few additional modules to help test as well.

I searched all the other action options and I can't find the modules (searched every entry in calibre actions as well).

I'm using action chains version 1.18.5

Bigwoof
bigwoof is offline   Reply With Quote
Old 03-11-2023, 01:06 AM   #1047
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,196
Karma: 1995558
Join Date: Aug 2015
Device: Kindle
After creating the module, the action "Modified Chain Caller" should appear in the dropdown list under "User Actions" node. You don't see that?
capink is offline   Reply With Quote
Old 03-11-2023, 01:20 AM   #1048
bigwoof
Connoisseur
bigwoof began at the beginning.
 
Posts: 88
Karma: 18
Join Date: Jun 2020
Device: Boox Note Air3, don't use my kindle anymore
Quote:
Originally Posted by capink View Post
After creating the module, the action "Modified Chain Caller" should appear in the dropdown list under "User Actions" node. You don't see that?
Hi,

I can't find a "User Actions" node in the drop down list under actions when you add a new row in a new action chain.

Bigwoof
bigwoof is offline   Reply With Quote
Old 03-11-2023, 01:53 AM   #1049
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,196
Karma: 1995558
Join Date: Aug 2015
Device: Kindle
OK. I think I figured out what the problem at your end is. So I modified the module to explicitly set is_experimental to False.

Please try again with the new code (updated the previous post).

Last edited by capink; 03-26-2023 at 04:32 PM.
capink is offline   Reply With Quote
Old 03-11-2023, 02:34 AM   #1050
bigwoof
Connoisseur
bigwoof began at the beginning.
 
Posts: 88
Karma: 18
Join Date: Jun 2020
Device: Boox Note Air3, don't use my kindle anymore
Quote:
Originally Posted by capink View Post
OK. I think I figured out what the problem at your end is. So I modified the module (copy below) to explicitly set is_experimental to False.
Thanks! That worked! worked fine on a test of 40 books. running it on 2,000 now :-)



EDIT:

2,000 worked as well. slow and steady wins the race!! no significant CPU or memory build up either. Thank you once again. Action chains are fantastic as a way to change things programatically. And this mechanism allows the chains to run on every book in large libraries..

This would be an excellent addition to the list of examples you already have.. :-)

Note: The problem I was trying to solve with my chains was to reduce the set of tags in my books from ~50,000 to under 200. Most of these tags come from ao3 authors going mad and myself not caring previously.. I finally bit the bullet and wrote some chains to extract only things I care about into tags.. The goal is to make it much easier to find things I care about on my various readers (Android and Kobo) as the number of tags, currently, overwhelms the readers on both those devices and the reader software I use (moon+, lithium, native kobo reader) on those devices don't really look at other attributes beyond title, author, series, and tags.

my other solution, before the "process files in chunks" solution provided by capink, was to edit the sqlite db directly using a sqlite admin tool. However, this has its own issues as the metadata is scattered across multiple tables (needing the right set of join commands) and my SQL foo is weak..

Thanks again!

Bigwoof

Last edited by bigwoof; 03-12-2023 at 12:33 AM.
bigwoof is offline   Reply With Quote
Reply


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Editor Plugin] Editor Chains capink Plugins 106 06-17-2025 05:36 PM
Action Chains Resources capink Plugins 77 06-16-2025 12:45 PM
[GUI Plugin] Noosfere_util, a companion plugin to noosfere DB lrpirlet Plugins 2 08-18-2022 03:15 PM
[GUI Plugin] Save Virtual Libraries To Column (GUI) chaley Plugins 14 04-04-2021 05:25 AM


All times are GMT -4. The time now is 10:49 PM.


MobileRead.com is a privately owned, operated and funded community.