View Single Post
Old 01-27-2011, 06:44 PM   #1
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
[GUI Plugin] Search the Internet

This plugin was designed to allow quick navigation from the selected books in the library view in Calibre to a choice of websites in your web browser.

By default websites such as FantasticFiction, Amazon, Google and Wikipedia are assigned, however around 100 more website links are included for you to select from and you can add/remove your own to this list.

I mainly use this plugin to look for better quality metadata than that available from the default Calibre sources, however it is also useful to get detailed reviews, identify missing books in a series, accurate publication dates, find similar authors etc.

Main Features of v1.7.4
  • Launch website searches for books matching author, title, isbn or any other book metadata field
  • Can assign multiple pages to open at once in a special group
  • Easy to use configuration dialog accessed via right-click menu or Preferences->Plugins
  • Over 100 predefined website urls for you to choose from, plus the ability to add your own.
  • Supports HTTP POST searches for websites that cannot use a querystring
  • Menu is fully customisable to change ordering, add submenus, keyboard shortcuts and images
  • Import and export capability for backup or sharing with other users
  • Help file web page available from the configuration dialog

Special Notes:
  • Requires Calibre 0.8.18 or later.

Installation Steps:
  1. Download the attached zip file and install the plugin/restart Calibre/add to context menu as described in the Introduction to plugins thread.
  2. A menu such as in the first screenshot attached to this post should appear. If you desire no changes to the list of websites you need do nothing further.
  3. If you want to add further websites, keyboard shortcuts, reorder/remove from this list, change menu names etc then this is possible by clicking on "Customize plugin" in the context menu. This is also available via the Preferences->Plugins dialog.

Tutorial for Adding a Website Link:
Spoiler:
In this example we will add a link to a website not currently in the plugin (www.fictionwise.com)
  1. The first step is to identify how the website accepts search information. Just try using the search box on the website, taking note of what appears in your browser bar when the results are displayed.
  2. If the url displayed in the browser bar contains a "querystring" then this will be the simplest website to add. Take for example www.goodreads.com. When I type "hunger games", the url displayed is http://www.goodreads.com/search?q=hunger+games. So we know that a simple substitution of plugin tokens like {title} in place of "hunger+games" would give us a search url we could re-use. This is called an HTTP GET type of query.
  3. If however the url displayed does not contain the text you searched, this means an HTTP POST query is taking place. This means the data we search with must be sent using code in the web page and requires a bit more work. If we search for the author "lyn gala" on fictionwise.com, the url displayed is http://www.fictionwise.com/servlet/mwsearch. So this tells us this website falls into this latter category.
  4. The next step will be to work out what we need to pass to the website. If you are using HTTP GET, you can usually just experiment with the website, trying different search dropdowns it has and taking note of how the url of the results changes depending on whether you search by author, title etc. Every website handles this differently.
  5. For HTTP POST websites we must use view source page from your browser. We will be looking for "<form " in the html displayed for the special code that submits the search data. In fictionwise.com we find "<form name=searchform..." near the top. We can see in the "action" that data will be sent to "/servlet/mwsearch". which matches the url we saw displayed in the browser. We can see a bunch of <input> elements including searchstring which will contain our text, a <select> element called searchtype with values like title and author. There are also some hidden <input> tags for action and template
  6. We need to put together our name=value pairs of data to supply, using either fixed values specific to the website, or dynamic values using plugin tokens like {title} to pass the title of the selected book. In this case to search on fictionwise.com for an author we know we will want to set searchtype=author and searchstring={author}
  7. We are now ready to start configuring this in the plugin for testing. In the search the internet plugin, choose Customize plugin and click on the green plus button to add a new row.
  8. Type a value into the Title column, like "Fictionwise for Author"
  9. In the Method column on the far right, we will choose POST or GET. At first glance, it seems like we should use POST in this particular case.
  10. In the Url column, we need to put where we will send our search query and what data to send. We know that the url will start with "http://www.fictionwise.com" and according to the action attribute on the form we must add "/servlet/mwsearch" to this. To be able to pass our pairs of name=value data we will then use the same syntax as if we were doing an HTTP GET query of "?name=value1&name2=value2" etc.
  11. So in this case the full url we will use is:
    Code:
    http://www.fictionwise.com/servlet/mwsearch?searchstring={author}&searchtype=author&action=view&template=searchresults.htm
  12. Right now we have enough to begin testing. Right-click on our new menu row in the configuration dialog, and choose "Test url" and verify the website is displayed as you would expect. If the website has no matches for that particular test data, use the "Test url using..." menu option instead.
  13. As an experiment, try flipping the Method to GET instead of POST. Just because the website when you searched didn't use/display such a URL indicating a GET query, sometimes they will still go ahead and accept it. The advantage if it does is that will make your navigating to it slightly faster. In the case of fictionwise.com, we find the above url does actually work, so now we can keep the Method column set to GET.
  14. Our final common step for the website is to specify an image to appear in the menu for this item. The most common source is what as known as a "favicon" and is the icon you see in your web browser next to the url when you go to the website or bookmark a page.
  15. Right-click on our row in the configuration dialog and choose "Add image...". In this case fictionwise.com has a favicon so we will use the plugin to retrieve it. So in the dialog select "From web domain favicon" and in the box next to it type "www.fictionwise.com". In the "Save as filename" box, give it a unique name, like "fictionwise" and click ok.
  16. In the Image column, select "fictionwise.png" in the dropdown that should now be present.
  17. At this point we have a fully working menu item. Move it up and down the list as desired and specify any other options like whether to appear in a submenu, a keyboard shortcut etc.
  18. Add additional variations as required for searching for a title, for a book (author and title), for ISBN etc.

The final step if desired is to make your urls available for other users on MobileRead.
  1. Select your fictionwise menu items together in the plugin configuration dialog.
  2. Right-click and choose Export. Type a name to save this as. This will save the configuration information and images into a zip file.
  3. Attach the zip file to a forum post in this thread. Job done.
You can find the resulting zip file from this tutorial attached to this post. If you think all of the above sounds like lots of work, then you may appreciate how much effort I have spent pre-configuring over 100 URLs in this plugin already for you

Paypal Donations:
  • If you find this or any of my other plugins useful please feel free to show your appreciation. I have spent many hundreds of unpaid hours in their development and support so any encouragement for me to continue is appreciated!

Version History:
Spoiler:

Version 1.7.4 - 28 Jul 2014
Support for upcoming calibre 2.0

Version 1.7.1 - 24 Sep 2011
When doing HTTP POST queries (like Fiction DB) do not encode the passed query values, just escape them instead.

Version 1.7.0 - 11 Sep 2011
Switch the formatter used for resolving foreign names as the "new" SafeFormat is broken for this plugin's purposes.
Upgrade to support the centralised keyboard shortcut management in Calibre

Version 1.6.6 - 10 Aug 2011
Change the location of SafeFormat class which was removed from Calibre for release 0.8.14

Version 1.6.5 - 08 May 2011
Change webbrowser launching to use Calibre's wrapper for the default browser for better Linux support

Version 1.6.4 - 09 Apr 2011
Support skinning of icons by putting them in a plugin name subfolder of local resources/images

Version 1.6.3 - 05 Apr 2011
Ensure non-valid characters are removed when encoding

Version 1.6.2 - 04 Apr 2011
Correct version number to reflect 1.6.1

Version 1.6.1 - 04 Apr 2011
Fix bug of open group menu having incorrect icon_name argument

Version 1.6 - 03 Apr 2011
Rewritten for new plugin infrastructure in Calibre 0.7.53

Version 1.5.1 - 28 Jan 2011
Bug-fix for swapping rows after addition of GET/POST column
Display clickable button on HTTP POST page if javascript disabled

Version 1.5 - 28 Jan 2011
Use the Calibre template processor to support other metadata fields
Support HTTP POST websites like FictFact
Archive import export use standard zip extension for ease of uploading to forums
Ensure config window is parented to prevent multiple Calibre windows in taskbar

Version 1.4 - 24 Jan 2011
Split into separate code files with proxy architecture
Add a configuration dialog to replace tweaks file
Make menus dynamically recreated without restarts
Build in library of websites for users to choose from
Ensure author always passed in FN LN format

Version 1.3 - 31 Dec 2010
Support Calibre 0.7.34 feature of plugin images retained within the zip file

Version 1.2 - 31 Dec 2010
Added support for encoding for passing foreign language names to Amazon etc

Version 1.1 - 04 Dec 2010
Added support for multiple row selections to launching multiple searches

Version 1.0 - 28 Nov 2010
Initial release of Search The Internet plugin
Attached Thumbnails
Click image for larger version

Name:	Screenshot_1_DefaultMenu.png
Views:	2418
Size:	33.6 KB
ID:	65611   Click image for larger version

Name:	Screenshot_2_ComplexMenu.png
Views:	1941
Size:	56.7 KB
ID:	65612   Click image for larger version

Name:	Screenshot_3_Configuration.png
Views:	1493
Size:	143.7 KB
ID:	65613   Click image for larger version

Name:	Screenshot_4_Add_Image.png
Views:	1608
Size:	25.4 KB
ID:	65614  
Attached Files
File Type: zip Search The Internet-qt5.zip (83.2 KB, 4166 views)

Last edited by kovidgoyal; 07-28-2014 at 03:29 AM. Reason: v1.7.4 Released
kiwidude is offline   Reply With Quote