Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 04-12-2013, 11:02 AM   #1
Satas
Member
Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.
 
Posts: 14
Karma: 3088
Join Date: Apr 2013
Device: none
Make a simple Plugin for Full Text Search using Recoll

Hello, I have decided to make a very simple but useful plugin for calibre.

The idea is simple, I use recoll -t option to search in "Calibre Library" directory for documents, which contain a phrase you want to find.

The Plugin will have a simple structure.

1. Button to update the recoll index
2. a place to input the text you want to search (possible also with AND and OR)
3. Run the search

It is not a accurate method but I am not a programmer so just want to combine this two programs in a way, I can use them best.

I search for the (id) tag in the results from recoll stdout and want to update the gui table with the books with this ids.

Now I have some questions:

1) when do this numbers change or do they change after a book is deleted or added or something else (a not so important question because you can just update the recoll database every time you make some changes).

2) I want to check the actual filter (books actually shown in calibre) and compare the ids to my search from recoll. Then I can display only the results matching both filters and you can combine calibre filter and recoll. For this I want to know where to find all the information how to use different methods for example for the LibraryDatabase2 Calss. I even cant find which methods exist.

Something in that sort of code is the thing I need:
Code:
        
        # Mark the records with the matching ids
        self.db.set_marked_ids(matched_ids)

        # Tell the GUI to search for all marked records
        self.gui.search.setEditText('marked:true')
        self.gui.search.do_search()
The only thing I have looked at is the example plugin and some .py of the db folder. I have no idea how to get the informations.

Well, if someone could help, it would be very nice.

PS: For someone who is a better programmer recoll has a python API, too. Maybe combine this two programs that way could solve the missing feature of having a full text search finally.

Thanks
Satas is offline   Reply With Quote
Old 04-17-2013, 09:52 AM   #2
Man Eating Duck
Addict
Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.Man Eating Duck juggles neatly with hedgehogs.
 
Posts: 253
Karma: 69784
Join Date: May 2006
Location: Oslo, Norway
Device: Kobo Aura, Sony PRS-650
The plugin Quality Check surprisingly has a very handy full-text search for epubs only. It doesn't maintain an index, but it does mark all entries with hits and restricts the window to these. You could have a look at the source and see if it has any useful hints. It might also be useful to you as it is, if your documents are epubs or it is feasible to convert them.
Man Eating Duck is offline   Reply With Quote
Old 04-17-2013, 10:03 AM   #3
Satas
Member
Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.
 
Posts: 14
Karma: 3088
Join Date: Apr 2013
Device: none
Thanks for the answer. Unfortunately all my books are pdfs (Springer ebook from the uni) so the plugin itself can not be helpful for me. But maybe I find the right thinks, which I can use in my plugin.
Satas is offline   Reply With Quote
Old 04-17-2013, 07:25 PM   #4
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,963
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
@Satas - I am interested in this because content searching is something I use extensively, I run Calibre on Windows, but I have colleagues with OS/X.

My understanding is that Recoll is only available for Linux, according to Calibre's Usage Stats 98% of Calibre installations are on Windows or OS/X.

Windows (since Vista) and OS/X have usable built-in content search facilities, i.e. Windows Search (WS) and Spotlight.

Have you considered DocFetcher, it runs on Windows, OS/X and Linux and its open source. Its written in Java so it doesn't have the same code level affinity with Calibre that Recoll would have.

Recoll and Spotlight DO index common ebook file types such as ePUB and Mobi, DocFetcher, and WS DO NOT index those file types. I overcome this deficiency by creating and RTF or TXT format via conversion.

However most of the documents I (and my colleagues) want to search originate from PDF, MS or Open Office files. The only source that I use regularly that provides anything else is McKinseys, they've recently started publishing some reports as ePubs alongside their PDF's.

This is what I am now doing in Windows.
  • I search the relevant Calibre Library folder with Windows Search, this presents me with a list of files that match the search term
  • I copy/paste the paths of the results into my text editor (Notepad++), which I massage into csv form with Author and Title columns.
  • I then load the csv into Import List plugin, which it matches to the Calibre database and from there I can create a reading list, add a tag to the books - basically do whatever I want.
Since last night my colleague is doing the same thing on her Macbook using Spotlight.

BR

Last edited by BetterRed; 04-17-2013 at 07:29 PM.
BetterRed is offline   Reply With Quote
Old 04-17-2013, 07:43 PM   #5
Satas
Member
Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.
 
Posts: 14
Karma: 3088
Join Date: Apr 2013
Device: none
Hello,

well, maybe the number of 98 percent is a big one, but for me it is of no importance. I can not write for windows. It is like asking someone to hurt himself. It is only my personal opinion.

I now have found the way I can integrate recoll very good.

I add a new column with the calibres id (also stored in the name of the directory). So now, after I make a search with recoll I can get a string like:

Code:
#cid:=34 or =56 or =76
and just use it for search these books in calibre.

I now will try to make a plugin for that.

docfetcher or something other is not an alternative as an external program for me, because they do not have terminal (cmd think on unix) access, so you have to start the program with a gui.

The other thing is, that I have bad experiences with java so I do not like programs written in it.

I can tell you, how to make an own column with the ids, maybe, if you use something else you could create a string containing the ids and copy/paste it into calibre.

Preferencies -> add you own columns
Lookup name: cid
Column heading: CID
Column type: Column built form other columns
Template: {id}

Hope, that can help.
Satas is offline   Reply With Quote
Old 04-17-2013, 09:42 PM   #6
Satas
Member
Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.
 
Posts: 14
Karma: 3088
Join Date: Apr 2013
Device: none
Have a small difficulty and have written a new thread: http://www.mobileread.com/forums/sho...57#post2485757

Would be glad to have some help.
Satas is offline   Reply With Quote
Old 04-17-2013, 09:48 PM   #7
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,963
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
I'm with you on Java, I wont it let onto my system.

I like your idea of doing the Search in Recoll and then using its results to retrieve the books in Calibre. Integration at the backend is usually easier and less troublesome than it is at the front end. I call it 'loose coupling'.

Why couldn't your plugin use the id's it gets from Recoll (i.e. #cid:=34 or =56 or =76) to access the metadata.db directly. My understanding is that the id (which Calibre puts at the end of the book directory names) is the primary key of the book table - see attachment.

Good luck - BR
Attached Thumbnails
Click image for larger version

Name:	Capture.JPG
Views:	127
Size:	120.0 KB
ID:	104543  
BetterRed is offline   Reply With Quote
Old 04-17-2013, 10:04 PM   #8
Satas
Member
Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.
 
Posts: 14
Karma: 3088
Join Date: Apr 2013
Device: none
Well, I could if I were a good programmer and would have a lot of time. But actually I only search for an easiest way to solve my own problem.

But with the thought in mind, that someone afterwards could take my little work and improve it.

So actually, I will first do it with the normal filter and bring it to work for me, then we will see.

And yes, I now the id comes from the database, but databases are a theme I never had to deal with.
Satas is offline   Reply With Quote
Old 04-19-2013, 10:38 PM   #9
Satas
Member
Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.Satas could sell banana peel slippers to a Deveel.
 
Posts: 14
Karma: 3088
Join Date: Apr 2013
Device: none
The plugin is ready to use: http://www.mobileread.com/forums/sho...d.php?t=211137
Satas is offline   Reply With Quote
Old 07-20-2013, 05:15 PM   #10
misterstudebaker
Junior Member
misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.misterstudebaker could sell banana peel slippers to a Deveel.
 
Posts: 3
Karma: 3088
Join Date: Jan 2012
Device: none
How about Lucene from Apache? It's used in Dropout. I'd love to see Dropout rolled into Calibre, but in lieu of that, i would think a plug in that uses Lucene would rock. Here's Lucene:
http://lucene.apache.org/core/
misterstudebaker is offline   Reply With Quote
Reply

Tags
database, developement, plugin, search

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Full Text Search? silentguy Calibre 4 02-22-2012 04:03 PM
Feature request: full text search Laisvunas EPUBReader 3 04-03-2011 12:47 PM
Full Text Search Engine Fat Abe General Discussions 1 09-21-2010 06:30 PM
Ebook management software with full text search SadE Reading and Management 4 05-23-2010 07:02 AM
Google Book Search to search full-text books online Bob Russell Deals, Freebies, and Resources (No Self-Promotion) 1 08-19-2006 01:13 PM


All times are GMT -4. The time now is 03:19 AM.


MobileRead.com is a privately owned, operated and funded community.