Plugin Best Practices

[sbenz]

I am developing a 'Bad Word' plugin to scan books for words the reader would rather not encounter. Based on the plugin response, the reader can choose to keep the book or delete it. The approach is simple:
1. BWFile.txt - a plugin resource that contains a list of bad words provided by the plugin but editable by the user
2. On scanning a book the plugin generates two items:

a. <bookid>_bwstats.txt - for each bad word in BWFile.txt it stores the number of times the bad word was encountered in the book

b. bwstats_M_N tag - adds this tag to the book indicating the book was scanned and had M unique bad words and N total bad words (this provides the reader a way to know if the book was scanned and a summary of the results)

3. Scanning is triggered two ways:

a. On adding a book (configurable)

b. On demand by the user

4. Review Results: the use may filter on books with the tag and via the plugin bring up the <bookid>_bwstats.txt for review before deciding whether to keep or delete the book

My questions:
1. Maintaining BWFile.txt on plugin update. If the user has edited the BWFile.txt (or preferences for that matter), how is it kept from being overwritten by an update to the plugin? The plugin installation is performed by Calibre and I haven't seen any documentation on how to keep existing plugin resources for use by the updated plugin.
2. Storing <bookid>_bwstats.txt:

a. In same folder as the book? Ideally it should be stored in the same folder as the book since the data is directly related to the book. If the book is removed from the library this file would also be removed. Is this a good practice? Perhaps it should be stored as meta-data, but at this point I don't know how I would do it.

b. How do I obtain the path to the folder containing the book?

1) db.field_for('path', bookid) returns the path relative to the Calibre library, but I haven't found an environment variable or method that specifies the Calibre library path.

2) db.format_abspath(bookid, 'epub') returns the absolute path to the book but requires a specific book format. What if there isn't an epub format? Furthermore, the API warns against using this approach for thread safety.

3. How do I create a dual use plugin, that is, one that acts like a FileTypePlugin (triggered when adding a book) and one that acts like a UserInterfacePlugin (triggered by user action). The core of the plugin is the same in both cases. The FileTypePlugin adds derivation from the FileTypePlugin class, has numerous 'on_...' instructions, and uses the 'def run(...)' method to be executed by Calibre. The UserInterfacePlugin derives from the InterfaceActionBase would call a different method and has added job control code. Can I combine the two into one plugin or should these be separate plugins? Are there plugin examples already doing this?

Any guidance would be appreciated!

[davidfor]

Originally Posted by sbenz

I am developing a 'Bad Word' plugin to scan books for words the reader would rather not encounter. Based on the plugin response, the reader can choose to keep the book or delete it. The approach is simple:
1. BWFile.txt - a plugin resource that contains a list of bad words provided by the plugin but editable by the user
2. On scanning a book the plugin generates two items:

a. <bookid>_bwstats.txt - for each bad word in BWFile.txt it stores the number of times the bad word was encountered in the book

b. bwstats_M_N tag - adds this tag to the book indicating the book was scanned and had M unique bad words and N total bad words (this provides the reader a way to know if the book was scanned and a summary of the results)

3. Scanning is triggered two ways:

a. On adding a book (configurable)

b. On demand by the user

4. Review Results: the use may filter on books with the tag and via the plugin bring up the <bookid>_bwstats.txt for review before deciding whether to keep or delete the book

I can't say I have need for this, but, I'll probably try it and see how "bad" some of my books are And of course, I can see someone using it to select the book to read.

My questions:
1. Maintaining BWFile.txt on plugin update. If the user has edited the BWFile.txt (or preferences for that matter), how is it kept from being overwritten by an update to the plugin? The plugin installation is performed by Calibre and I haven't seen any documentation on how to keep existing plugin resources for use by the updated plugin.

I can think of a few ways:
- Calculate the md5/crc or whatever for the files and only replace if it hasn't changed. Or prompt the user to decide.
- Use two files. One is the defaults that is always replaced. The other is the users file. The decision then is whether the use file is a complete replacement, or just extra words for that user.
- Don't replace the file, but have a way to download it from somewhere with a notification when it has changed.

2. Storing <bookid>_bwstats.txt:

a. In same folder as the book? Ideally it should be stored in the same folder as the book since the data is directly related to the book. If the book is removed from the library this file would also be removed. Is this a good practice? Perhaps it should be stored as meta-data, but at this point I don't know how I would do it.

b. How do I obtain the path to the folder containing the book?

1) db.field_for('path', bookid) returns the path relative to the Calibre library, but I haven't found an environment variable or method that specifies the Calibre library path.

2) db.format_abspath(bookid, 'epub') returns the absolute path to the book but requires a specific book format. What if there isn't an epub format? Furthermore, the API warns against using this approach for thread safety.

As you are running this within calibre against a book in the library, you can do either of these easily. Storing the data in a custom column is easy and is done in a lot of plugins. Both of my plugins do it, plus the Count Pages (which is where I cribbed the code from), Reading List, Goodreads Sync and others. And look at Kovid's post in https://www.mobileread.com/forums/sho...d.php?t=249522.

I haven't added a file to the library, but there should be simple way to add a format to the current book. That means you don't care about the path. You could add the file with your own extension then associate that in the OS with the text editor. Or add a "View stats" option to the plugin menu that displays the file.

3. How do I create a dual use plugin, that is, one that acts like a FileTypePlugin (triggered when adding a book) and one that acts like a UserInterfacePlugin (triggered by user action). The core of the plugin is the same in both cases. The FileTypePlugin adds derivation from the FileTypePlugin class, has numerous 'on_...' instructions, and uses the 'def run(...)' method to be executed by Calibre. The UserInterfacePlugin derives from the InterfaceActionBase would call a different method and has added job control code. Can I combine the two into one plugin or should these be separate plugins? Are there plugin examples already doing this?

Kovid will probably have a better way, but, what comes to mind is two plugins. One just uses a classes and methods in the other.

[kovidgoyal]

1. You store persistent data, just like any other settings, in the config directory, they are not touched by install/uninstalls of plugins. See http://manual.calibre-ebook.com/crea...of-your-plugin

You wont store it as a .txt file, instead it will be stored as a string in a JSON file.

2. Dont store book specific data as a file. Store it in the database. Using db.new_api.add_custom_data and db.new_api.get_custom_data

Naturally this will only work in a user interface pugin, which has access ot the db

3. Use two plugins. You can technically get fancy and have your user interface plugin install the file type plugin on initialization, but that is a hack and likely to break in the future.

[sbenz]

Thanks for the quick reply. Still some questions:

Originally Posted by kovidgoyal

1. You store persistent data, just like any other settings, in the config directory, they are not touched by install/uninstalls of plugins. See http://manual.calibre-ebook.com/crea...of-your-plugin

You wont store it as a .txt file, instead it will be stored as a string in a JSON file.

Thanks - makes sense. I suppose then the plugin should provide a way for the user to revert to the default bad word list.

Originally Posted by kovidgoyal

2. Dont store book specific data as a file. Store it in the database. Using db.new_api.add_custom_data and db.new_api.get_custom_data

Naturally this will only work in a user interface pugin, which has access ot the db

3. Use two plugins. You can technically get fancy and have your user interface plugin install the file type plugin on initialization, but that is a hack and likely to break in the future.

1. Storing book specific data in db versus a file. Normally I would agree with you that any meta-data should be in the db especially if it is used for sorting or searching, but this seems more like an attachment to the book that the db could reference rather that store. The data consists of a list of ('bad word', bad_word_count) tuple values and could be "large" depending on the size of the bad word list. Is it worth cluttering up the db with what could become a rather large amount of seldomly used information? There will be a bad word tag added to the book that the user can search on and if he so desires can then open up the list of bad word counts in a viewer.

2. Using: db.new_api.add_custom_data and db.new_api.get_custom_data. I would like to learn how to use this anyway. I have seen the documentation, (add_custom_book_data(name, val_map, delete_first=False) but don't understand what a value map is. Is this just a string? And is <name> any name of my choosing? (unique of course)

3. FileTypePlugin and db. Above you stated that I would need an InterfacePlugin to get access to the db. I presume you only meant if the user was wanting to see something since the FileTypePlugin would need to db access as well and I believe that is possible from the examples I have seen.

BTW - you have provided a real service with Calibre. It is a wonderful tool and I have already loaded it on multiple friends systems! I salute you.

[kovidgoyal]

You cannot store any files in the calibre library unless you are willing to give them their own extension and make them a format. And that will just confuse your users. Store your wordlist in the db, your life will be a lot simpler. If you put random files in calibre library folders they will be deleted automatically if the user chanes title or author and check library will complain about them.

Quoting the documentation:

A val_map is just a python dictionary, like this:

{book_id:"some value", book_id2:"other value"}

Yes, I was refering to providing an interface for your users to edit the stored data, which can happen only in a user interface plugin.

[sbenz]

Thanks for the quick reply!

Originally Posted by davidfor

I can't say I have need for this, but, I'll probably try it and see how "bad" some of my books are And of course, I can see someone using it to select the book to read.

This plugin is not for everyone! I have a rather eclectic taste and read a wide variety of books and can tolerate a lot. I will use it some, but I have friends who will use it a lot. And since the bad word list is customizable, you can use it as you see fit.

Originally Posted by davidfor

I can think of a few ways:
- Calculate the md5/crc or whatever for the files and only replace if it hasn't changed. Or prompt the user to decide.
- Use two files. One is the defaults that is always replaced. The other is the users file. The decision then is whether the use file is a complete replacement, or just extra words for that user.
- Don't replace the file, but have a way to download it from somewhere with a notification when it has changed.

Good ideas - Kovid gave the answer I was looking for - put the data in a json file in the config directory - this directory is not overwritten during plugin updates. May still have to use your ideas to determine if the data has changed and give the user a "revert" choice.

Originally Posted by davidfor

As you are running this within calibre against a book in the library, you can do either of these easily. Storing the data in a custom column is easy and is done in a lot of plugins. Both of my plugins do it, plus the Count Pages (which is where I cribbed the code from), Reading List, Goodreads Sync and others. And look at Kovid's post in https://www.mobileread.com/forums/sho...d.php?t=249522.

Due to the potentially "large" size of this data, storing it in the db could be unnecessarily cluttering it, especially since it will seldom be used. See my response to Kovid. I will look at the examples you provided since I may still need them - thanks.

[BetterRed]

The calibre editor provides a facility to maintain and import user defined 'dictionaries', they are stored in %calibre_config_directory%\dictionaries\prefs.json .

I wonder if sbenz's Bad Words PI could leverage that feature. Assuming different Bad Word 'dictionaries' would be used for different physical libraries then only the dictionary name need be stored in the library database.

Another advantage of leveraging the existing user dictionary facility might be that it is language aware.

Aside :

I doubt that storing the bad word list in the database would have any noticeable impact on performance. As for cluttering, one of my databases has 31 tables, 61 indices, 13 views and 46 triggers. I assume a bad_word_list would be a row in the preferences table and an update to the preferences autoindex, I can't see how that could be regarded as 'cluttering' the database.

One of my user dictionaries has ~6800 words, it weighs-in at 105MB, I'm pretty certain that SQLite can handle something of that size and much larger.

BR

[sbenz]

Originally Posted by BetterRed

The calibre editor provides a facility to maintain and import user defined 'dictionaries', they are stored in %calibre_config_directory%\dictionaries\prefs.json .

I wonder if sbenz's Bad Words PI could leverage that feature. Assuming different Bad Word 'dictionaries' would be used for different physical libraries then only the dictionary name need be stored in the library database.

Another advantage of leveraging the existing user dictionary facility might be that it is language aware.

I was hoping this capability existed since I would need to share the Bad Word dictionary between a FileTypePlugin and a UserInterfacePlugin per Kovid's comments.

Using multiple dictionaries occurred to me also for other reasons. I will explore this more after it is running.

Originally Posted by BetterRed

I doubt that storing the bad word list in the database would have any noticeable impact on performance. As for cluttering, one of my databases has 31 tables, 61 indices, 13 views and 46 triggers. I assume a bad_word_list would be a row in the preferences table and an update to the preferences autoindex, I can't see how that could be regarded as 'cluttering' the database.

One of my user dictionaries has ~6800 words, it weighs-in at 105MB, I'm pretty certain that SQLite can handle something of that size and much larger.

Apparently I have not been very clear regarding my data storage concerns - and they may not be a problem. The issue is not storing the BW dictionary, but rather the results for each book, BW_BookStats. Consider the following:

BW_Dict = list of bad words; stored only once in preferences
BW_BookStats = dict of (bw, count) pairs where count is number of occurrences of bw in the book.

1. BW_BookStats is per book. If BW_BookStats averages 10KB, a 10,000 book library now has 100MB of data. Since the user controls the size of the BW_Dict and the number of books, I don't know how large this might grow.
2. Reducing total BW_BookStats storage:
   • Zero counts are not stored
   • Only a few byte ActionStatus is stored for 'Clean' books and books marked 'DoNotScan'
3. BW_BookStats is likely used only once per book per library user to decide a course of action. Perhaps only a small bw summary should be provided and the BW_BookStats should just be rebuilt on demand. However, scanning can take a lot of time, especially on large books with a large bw list.

I really appreciate your input.
sb

[BetterRed]

@sbenz - Gotcha - yes I can see how a 'bad-word, count' list for every book could be regarded as 'cluttering' - especially if it's short-lived.

You probably did explain it well enough, but these days my brain disengages when I see 'programmer' lingo.

You can add one of any 'file type' to a book. So for example you can add an MP3 file to a book, calibre itself can't do much with an MP3 except hand it off to the OS in the hope it knows what to do with it. There's also an Open With PI that will hand off a file type to a designated program

So you could write the 'bad-word, count' list for a book in calibre's current temp folder with a name of <some random string or other>.bwlist, and then add that file to the book. You would then have a BWLIST 'format' in the book, calibre will then take care of it - ie not lose it if a folder is moved or renamed - but like an MP3 it won't be able to do much with it. Some alternatives for the internal representation that spring to mind include - json, csv, serialised.

BR

[sbenz]

I have carefully reviewed each option for my Bad Words List plugin and none are satisfactory.

• Add a book extenstion to store just the bad word data (confuse the reader)
• Store bad word data for each book in the db (clutters the db)
• Store the bad word data for a book in the book itself (if multiple formats do I append to each one)
• Create bad word data, summarize into metadata tag, delete bad word data (requires user to rescan to see the actual data)
• Create a parallel folder structure outside of Calibre to contain the bad word data (difficult for the user to maintain)

Furthermore, if I widened my horizons, the bad word plugin would become just a Scan Word plugin that could operate on multiple lists (bad words, historical words, urban words, etc.). These might be useful to authors and individuals studying word usage in period books. Now instead of having just one list result per book, there could be several. Storing these in an Attachment folder would be easy and save significant space in the db.

The best solution appears to be an "Attachment" folder in each book folder that any plugin could drop files into and manage. Would it be difficult to enhance Calibre to include an "Attachment" folder in each book folder? Initially Calibre's only responsibility for this folder would be:

1. Move/copy the folder with the book whenever the book is moved/copied
2. Delete the folder when the book is deleted

I don't want to write up an enhancement request unless this is relatively easy to achieve.

[BetterRed]

How would you prevent a user not putting 'other' things in the Attachments folder ?

BR

[sbenz]

Originally Posted by BetterRed

How would you prevent a user not putting 'other' things in the Attachments folder ?

BR

I wouldn't. I had actually thought of use cases where a user might want to put things in this folder. And if they did, it would not affect the db, only the size of the 'book folder,' so there would be no harm. It would be more problematic if a user deleted a file from the folder put there by a plugin. The plugin would have to handle those cases.

SB

[BetterRed]

Originally Posted by sbenz

I wouldn't. I had actually thought of use cases where a user might want to put things in this folder. And if they did, it would not affect the db, only the size of the 'book folder,' so there would be no harm. It would be more problematic if a user deleted a file from the folder put there by a plugin. The plugin would have to handle those cases.

SB

@sbenz - maybe the plugin data could be put into a different sub folder eg ...\attachments\plugins or ...\plugins

However, there have been several requests to have an Attachments (Extras, Supplementary) folder within the book folders into which user could put things. Kovid has declined to do that, I can't recall his rationale in detail but at the time it seemed reasonable. Might have been related in part at least to deletions, and what happens if a file in Attachments is locked by another process when there's a move to be done etc. I suggest you ask Kovid for his thoughts.

My observation is that the file handling across different platforms is one of the more 'difficult' issues calibre has to address - case significance, access control, file locking, path lengths etc.

Which maybe brings you full circle to either having the word list stored in the database, or as a 'pseudo_format file', or... in unmanaged space.

Your main objection to the 'pseudo_format_file' was that seeing a file called 'Some Title - An Author.badwords' would confuse the user. But would that be any more confusing than pressing 'O' and seeing an Attachments folder containing a file of a similar name. Or coming to grips with the purpose of ORIGINAL_XXXX.xxxx files. Transparency is often the best way to avoid confusion.

The file wouldn't come into existence unless the user ran your PI, and presumably your PI would hook up to a custom column as a place to store the bad word count. And the user could colour that column based on the count, blue if more than 50, red if more then 20 etc

BR