[GUI Plugin] Search the Internet

[asalet_r]

thank u @kiwidude
but ı have already tried it, it doesnt work.
but I'm still working on it.

[copyrite]

I am having problems with the 'FictionDB for Book' option.

Did I muck up the code somehow or is it having issues?

Spoiler:

http://www.fictiondb.com/search/searchresults.htm?styp=6&author={author}&title={ti tle}&srchtxt=multi&sgcode=0&tpcode=0&imprint=0&pub group=0&genretype=--&rating=-&myrating=-&status=-

It returns:

The search results for: multi

uNothing found!

I am not having problems with the title and author search.

Thanks!

[kiwidude]

@copyrite - it seems that they have changed their website.

It's a pretty simple fix: Change the url to this:

http://www.fictiondb.com/search/searchresults.htm?styp=5&srchtxt={author}+{title}

And in the far right column change the "Method" from "POST" to "GET".

Alternatively use the same URL with a space where the + is between {author} and {title}, and keep the method as "POST". Both seem to work equally from my quick tests.

[copyrite]

That works - thanks, kiwidude!

[dFGJByjm4898IssG]

Using the OnePetro website to search for technical papers, the site address is:

http://www.onepetro.org/mslib/app/newSearch.do

And a manual address to get a paper is:

http://www.onepetro.org/mslib/app/Pr...ocietyCode=SPE

And the plugin address using the series_index is:

http://www.onepetro.org/mslib/app/Preview.do?paperNumber=SPE-{series_index:re(0$,)}-PA&societyCode=SPE

And this works.

My question is how to I extract the Title and other fields from the web page and place them in Calibre?

Any help will be greatly appreciated.

[kiwidude]

You can't with this plugin - its purpose is to display a web page in a browser, nothing more. So any "extraction" would be manually copy/pasting.

For what you want to do you need to write a metadata download plugin, such as those included with calibre like Amazon & Google or various examples in this forum like Goodreads, Fantastic Fiction, Webscription, etc etc.

[dFGJByjm4898IssG]

Thanks Kwidude.

It would have been nice for your plugin to have fields, one for title, author, comments etc. where one could use a regular expression to pickup the title etc.; similar to the way Calibre reads the metadata from the file names. So instead of file names it reads the source from the web page.

I guess I am going to have to learn python :-(

[kiwidude]

A metadata download plugin is one of the easiest types of plugins to write, but then I might be biased from having written so many of them. Most of their pain comes from having to continually update them as the website changes, which is annoying. Unfortunately your particular website requirement is very niche - you could well be the only calibre user in the world storing those particular documents, or else I would just knock it up for you. If you really can't face the learning curves and it means enough to you I can always be bribed via PM of course .

I wouldn't say that scraping data is a functional match for this particular plugin, but I understand what you are asking for. In fact another plugin I have written recently but not released yet does something similar in concept, scraping title/author data from pages using user configurable xpath and regex expressions. It wouldn't be useful to you for this, but I'm certainly well aware of the "fun" involved.

I did wonder ages ago about making a generic metadata scraping plugin which would allow a knowledgable end user to configure a website to pull data from specifying similar xpath/regex expressions. However you are talking very niche usage, and there would always be some sites where the generic approach could not work since every site is different. Then there is also the issues of interaction with the existing metadata download processes etc. It isn't impossible, but it just hasn't been asked for enough for someone to be motivated enough to write it.

[dFGJByjm4898IssG]

I agree I will probably be the only user of this plugin as it is so specialized. So I think I will try to do this myself first and see how it goes. Out of curiosity, if you were to write the plugin how much is the bribe :-) ? Remember there will be a few "issues" with this site as I have found out myself.

[dtrounson]

Hi Guys,
Just incase you have a lot of roleplaying books that need sorted, here are some settings for DriveThru RPG.

For Book - http://rpg.drivethrustuff.com/index.php?keywords={title}&x=0&y=0&author=&artist= &pfrom=&pto=
For Author - http://rpg.drivethrustuff.com/index.php?keywords=&x=0&y=0&author{author}=&artist =&pfrom=&pto=
Favicon Link = http://rpg.drivethrustuff.com/themes/dtrpg/favicon.ico (though that didnt work when I tried it, will probably just make the .ico a .png and add it manually)

I was having a hard time being motivated to edit all of my White Wolf stuff manually, and came across this page. In the short term Search the Internet will work, but I am actually VERY tempted to try writing some form of RPGBooksDB type page with an API, since its likely a community people would get behind

Anyway hope this helps someone

[kiwidude]

Thanks for sharing dtrounson. It is of course possible to export your settings into a little zip file that others can then import which makes it a little easier but the important part is figuring out the search syntax which you have done. HOpefully others will find it useful.

[UFS]

Hi,

I just installed this plug-in and noticed that when I go to configure the plugin the memory usage for calibre shoots up to nearly 2gig (1,802,652K in windows task manager)

before i noticed this i had added a couple of my own searches so i thought i might have messed it up, so i uninstalled and re-installed the plugin, but it still does it.

other than that it works great

any reason for this?

cheers,
chris

[Guba]

For those italians, this is the configuration line for searching on HOEPLI bookseller database:
http://www.hoepli.it/cerca/libri.aspx?query=%22{author}%22+%22{title}%22&ty=1

Enjoy

[Guba]

For the author of the plugin: it crashes all the time during configuration operation.
I'm using the last version 0.8.67

Any workaround ?

Guba

[theducks]

Originally Posted by Guba

For the author of the plugin: it crashes all the time during configuration operation.
I'm using the last version 0.8.67

Any workaround ?

Guba

Search The Internet is Not part of the Calibre didtribution, but a User controlled update. (Preferences: Get plugins.. :Installed ) I believe V1.71` is the latest (and it does not crash on XP when selecting 'configure')