Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > More E-Book Readers > iRex

Notices

Reply
 
Thread Tools Search this Thread
Old 04-15-2008, 10:25 AM   #1
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Multi-directory search

This is a spin-off from something else I'm playing with. The built-in iliad search will only search the current directory, and if I'm not mistaken will only search the title field, not the description.

This little script will allow you to search multiple directories and will grep the manifest file. This means you can put key words in the description field and it will match on those.

There is another script (contentlister entry) for deleting the results. Don't use the built-in delete because this deletes the original files rather than the symlinks. I think thomas posted a better way to do symlinks and I'll try using that later.

You enter the search phrase by changing the FIRST LINE of the description field of the search entry. At the moment this is "wibble", because that is what I was searching for when testing this. Use the label function to change the description, then click on the search entry.

So, in summary, these are the steps:
  1. get shell access if you do not already have it
  2. unzip search.zip somewhere, e.g. your PC
  3. copy the whole thing to somewhere on your iliad (anywhere will do)
  4. navigate to the search folder using the contentlister
  5. use the config entry to select where to search
  6. change the description field to your search expression
  7. click on the "search" entry in the contentlister
  8. when the search is complete you will see a "results" folder
  9. view your results in the results folder
  10. delete the results using the "delete" entry in the contentlister
I started doing this because I think I will soon have a way of tagging and taking snippets from PDFs and plan to store the tags in a tags file in the appropriate PDF container directory and snippets in a snippets file. I want to be able to search for snippets or tags.

The main limitations at the moment are:
  1. no way to search within search results
  2. you have to navigate to the search folder, you can't just get there quickly from anywhere
  3. inelegant way of getting user input

You can set the place to start searches to any location you like. Just add entries to the file config/search.rc

I've tested this a little on my USB key with about 800Mb of files and it did not fall over. Feedback welcome.

Edit: just made a couple of updates to zip file.
Edit: the limit on the number of results is not working. I'll have to look into this.
Edit: new version---quoted search expression to allow spaces
Edit (2008-05-01): new version that I think fixes the spaces in filename problem and adds a path config tool. Also fixed hit limit.
Edit (2008-05-01): Added snippet search tool
Edit (2008-08-11): Fixed CF search problem
Attached Files
File Type: zip search.zip (15.8 KB, 195 views)

Last edited by daudi; 08-11-2008 at 10:42 AM. Reason: New version 2008-08-11. Fixed CF search problem
daudi is offline   Reply With Quote
Old 04-16-2008, 08:13 AM   #2
yokos
Pac-Man catched my iLiad.
yokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with others
 
yokos's Avatar
 
Posts: 720
Karma: 2571
Join Date: Apr 2006
Location: Germany; next to Baltic Sea
Device: 1st gen iRex iLiad with 2nd ed. battery/case
It seems like spaces are not allowed in directory names.
Code:
Search string: Krause
grep: /mnt/cf/pdf/Foundations: No such file or directory
grep: of: No such file or directory
grep: GTK+: No such file or directory
grep: Development/manifest.xml: No such file or directory
[...]
./run.sh: 87: [[: not found
After changing dir's name from "Foundations of GTK+ Development" to "Foundations_of_GTK+_Development" it is working.


[EDIT] I had got a space bug in my jukebox mpd scripts, too.
yokos is offline   Reply With Quote
 
Enthusiast
Old 04-16-2008, 08:45 AM   #3
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Thanks! I've now quoted the grep so hopefully fixed the space bug.
daudi is offline   Reply With Quote
Old 04-16-2008, 09:08 AM   #4
yokos
Pac-Man catched my iLiad.
yokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with others
 
yokos's Avatar
 
Posts: 720
Karma: 2571
Join Date: Apr 2006
Location: Germany; next to Baltic Sea
Device: 1st gen iRex iLiad with 2nd ed. battery/case
Quote:
Originally Posted by daudi View Post
Thanks! I've now quoted the grep so hopefully fixed the space bug.
Not yet. It doesn't work with spaces.
[very OT] Battery of my iLiad is low & I don't have a power adapter next to mine.

Last edited by yokos; 04-16-2008 at 09:09 AM. Reason: doesnt != doesn't
yokos is offline   Reply With Quote
Old 04-28-2008, 05:53 AM   #5
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Quote:
Originally Posted by yokos View Post
Not yet. It doesn't work with spaces.
[very OT] Battery of my iLiad is low & I don't have a power adapter next to mine.
OK, I've looked into this a little more. To handle this nicely we need a newer version of busybox. The version on the iliad does not seem to support -print0 for find or -0 for xargs, but the current release does. This current release is version 1.10.1. Adam has released version 1.7.2 and I am not sure if that supports these options or not. I'll try to test it this evening. The alternative would be to use temporary files in the script instead. That would be a less elegant solution though.
daudi is offline   Reply With Quote
Old 04-29-2008, 09:03 AM   #6
yokos
Pac-Man catched my iLiad.
yokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with othersyokos plays well with others
 
yokos's Avatar
 
Posts: 720
Karma: 2571
Join Date: Apr 2006
Location: Germany; next to Baltic Sea
Device: 1st gen iRex iLiad with 2nd ed. battery/case
Quote:
Originally Posted by daudi View Post
OK, I've looked into this a little more. To handle this nicely we need a newer version of busybox. The version on the iliad does not seem to support -print0 for find or -0 for xargs, but the current release does. This current release is version 1.10.1. Adam has released version 1.7.2 and I am not sure if that supports these options or not. I'll try to test it this evening. The alternative would be to use temporary files in the script instead. That would be a less elegant solution though.
Oh, interesting.
It should be no deal to update busybox. Termial apps are dream targets to port compared to GUI apps with all the odds.
I think Adam hasn't enabled all features of busybox during configure to make the binary small.

EDIT:
Quote:
Originally Posted by README of busybox
Using busybox:

BusyBox is extremely configurable. This allows you to include only the
components and options you need, thereby reducing binary size. Run 'make
config' or 'make menuconfig' to select the functionality that you wish to
enable. (See 'make help' for more commands.)

Last edited by yokos; 04-29-2008 at 09:07 AM.
yokos is offline   Reply With Quote
Old 04-29-2008, 09:21 AM   #7
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Ahaaa. Very helpful, thanks. I'm not going to be able to look at this again for a few days, but this will save me time mucking around trying to figure out things that might not need figuring out. Thanks.
daudi is offline   Reply With Quote
Old 05-01-2008, 09:15 AM   #8
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
It turned out to be rather easy to make this work with the standard busybox so I have decided to do that rather than require people to upgrade their version of busybox. I have tested it and think it is working now with directories with spaces in the path and would be grateful it you could try it again Yokos.

I have also changed the way that search paths are configured. Instead of having to hard code the path in the script, I have now added a config contentlister entry that reads a file called search.rc. This file consists of one line per directory, a semi-colon and a description. The default contents on this file are:
Code:
/mnt/usb;USB
/mnt/cf;Compact flash
/mnt/free;All internal memories
You can easily add others to particular directories. With this you just click on the entry in the contentlister to cycle through the paths to search. The description field in shown in the contentlister is updated with the description of the current search path.

I've also fixed the number of hits limit code. This is currently set to 40 results. You can change this near the top of the script.

I'm updating the post at the top of this thread with this version.

Last edited by daudi; 05-01-2008 at 09:18 AM. Reason: Added note about hit limit code
daudi is offline   Reply With Quote
Old 05-02-2008, 11:37 AM   #9
Dabon
Connoisseur
Dabon doesn't litterDabon doesn't litter
 
Posts: 80
Karma: 111
Join Date: Apr 2008
Device: iliad V2; ipad 2-64GB,3G
Problem installing Search

Dear Daudi,

Good day!
Thank you for working so diligently on such great and useful tool.
I have tried to install this tool several times, but unsuccessfully. i am not getting the message " installation successful". I am able to switch the locations for research between internal memory, USB and CF, but the launching of research is unsuccessful, the iliad does not react when I click on search folder...
Any ideas or suggestions on what I could do to get it working?

Thanks a million in advance!

D.
Dabon is offline   Reply With Quote
Old 05-02-2008, 11:50 AM   #10
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Quote:
Originally Posted by Dabon View Post
Dear Daudi,

Good day!
Thank you for working so diligently on such great and useful tool.
I have tried to install this tool several times, but unsuccessfully. i am not getting the message " installation successful". I am able to switch the locations for research between internal memory, USB and CF, but the launching of research is unsuccessful, the iliad does not react when I click on search folder...
Any ideas or suggestions on what I could do to get it working?

Thanks a million in advance!

D.
There's nothing to install so there's no need for an "installation successful" message
I suspect it probably is running but using the default search pattern. I think I probably need to re-write the first post or write another one to make things clearer.

A key thing from my first post is:
Quote:
You enter the search phrase by changing the FIRST LINE of the description field of the search entry. At the moment this is "albert", because that is what I was searching for when testing this. Use the label function to change the description, then click on the search entry.
Have you used this to enter the search phrase? It is a kludge, I know, but seemed to me to be the simplest way of getting simple user input. Once you have entered your search phrase clicking on the search entry should run the search.

WARNING! At the moment the general search searches the entire manifest file. This means that if you are searching for "The Great Escape", for example, do not enter "esc" as the search phrase. "esc" will match "escape" okay, but it will also match "Description" which exists in (almost?) every manifest file.

BTW, I should also explain that the snippet-search relates to this thing.

If you have entered your search phrase correctly and it is still not working we'll have to delve a little deeper. Let me know how you get on.
daudi is offline   Reply With Quote
Old 08-08-2008, 01:25 AM   #11
engunneer
Member
engunneer began at the beginning.
 
Posts: 13
Karma: 10
Join Date: May 2008
Location: Pacific Northwest, US
Device: iRex Iliad
Quote:
Originally Posted by daudi View Post
If you have entered your search phrase correctly and it is still not working we'll have to delve a little deeper. Let me know how you get on.
I am having the same problem as Dabon.

The search folder is loaded in Newspapers/Programs/Search.
Newspapers is on the CF card (according to iLiad Settings/Archive Locations page)
The document I am searching for has a two line description. The only thing that may be odd is that the iLiad did not generate the manifest.xml. I am using another program that I wrote to generate it. It is located in documents/publications/[folder name]/ (also on the CF card)
I am editing the first description line of the search program using the tagging tool. In today's case, the search term is banding. The config program is switching modes just fine, but the search tool runs and no change is shown in contentlister (the screen does refresh). Is there some error log I can look at?

Thanks,

Last edited by engunneer; 08-08-2008 at 01:28 AM. Reason: forgot to mention path of expected result
engunneer is offline   Reply With Quote
Old 08-08-2008, 08:07 AM   #12
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Hi engunneer,

There's no logging at the moment, but I'll look into it. For now, some questions that might help to track down the source of the problem:
  1. When you click on the search entry (after setting the description to "banding"), does it seem to do anything at all? Do you get the two moving bars at the bottom of the screen that show that the iliad is thinking about something? (It did this for about 2 mins while searching 1Gb of files just now)
  2. Do you get a "results" contentlister entry? If so, your results are in there.
  3. What are the contents your config/search.rc? What is your currently selected config?
  4. You could try commenting out line 86
    Code:
    rm match.lst match.tmp
    and see if those files (match.lst and match.tmp) are created.
daudi is offline   Reply With Quote
Old 08-10-2008, 02:02 AM   #13
engunneer
Member
engunneer began at the beginning.
 
Posts: 13
Karma: 10
Join Date: May 2008
Location: Pacific Northwest, US
Device: iRex Iliad
Thanks for getting back to me.

1. The LED blinks and the White Loading bricks at the bottom do show for about 2 seconds. My CF card is 2GB and about half full. None of the bricks turn black.
2. No results at all (I do have the entry to delete results)
3. I am using the default one you have posted above. I have tried all of the settings, but Compact flash is the one that should work in this case.
4. the files are created, but both are empty.

If I can get this working, then I will try to get your snippet tool working (on windows), which looks very impressive. Then I will publish the project for the iLiad that I have been working on.

Update: Setting to "All internal memories" does at least run long enough to get one black box to fill.

At this point, I am suspecting that the manifet.xml of my target files are somehow wrong? I can try to upload an example doc tomorrow.

Last edited by engunneer; 08-10-2008 at 02:16 AM.
engunneer is offline   Reply With Quote
Old 08-10-2008, 09:41 AM   #14
daudi
Addict
daudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-booksdaudi has learned how to read e-books
 
Posts: 281
Karma: 904
Join Date: Oct 2007
Location: Newcastle upon Tyne, UK
Device: iRex iLiad, Psion 5MX, nokia n800
Quote:
Originally Posted by engunneer View Post
4. the files are created, but both are empty.

[...]

At this point, I am suspecting that the manifet.xml of my target files are somehow wrong? I can try to upload an example doc tomorrow.
match.tmp is a list of manifest files to check, so if that is empty it suggests that it is not even finding any manifest files. One thing that springs to mind now that you mention windows is case sensitivity. manifest.xml != manifest.XML and I think I have seen some windows programs cause the extension to be uppercase. The find command that I use only searches manifest.xml files (lower case extension).
Code:
find "$STARTDIR" -type f -name 'manifest.xml' > match.tmp
That would explain why match.tmp is empty. There's not actually much that can go wrong in this command: either the STARTDIR (starting directory) is wrong, or there are no files named 'manifest.xml'. Can you verify the case of the extension?
daudi is offline   Reply With Quote
Old 08-10-2008, 07:49 PM   #15
engunneer
Member
engunneer began at the beginning.
 
Posts: 13
Karma: 10
Join Date: May 2008
Location: Pacific Northwest, US
Device: iRex Iliad
I can verify the extension. Extensions in uppercase bug the heck out of me, and I also wrote the program that is generating them.

I also have some documents that were nout generated by my program, and I have manually tagged them on the iLiad, and the search program still doesn't find them. I have not tried to experiment with ssh access, but theoretically could if you think it will help me diagnose this. Thanks for taking the time to try and help.

Is there something I can hardcode instead of $STARTDIR, so that it is known good? should I try running it via mrxvt?

Thanks.

Last edited by engunneer; 08-10-2008 at 07:53 PM.
engunneer is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
SD Directory Limit single650 Sony Reader 3 03-09-2010 04:45 AM
I Can't add a directory takelu Calibre 2 02-05-2010 07:21 PM
directory structure Dopedangel Calibre 7 08-20-2009 06:17 AM
iLiad Directory Tree The GreatGonzo iRex Developer's Corner 2 09-18-2007 11:52 AM
Google Book Search to search full-text books online Bob Russell Deals, Freebies, and Resources (No Self-Promotion) 1 08-19-2006 12:13 PM


All times are GMT -4. The time now is 06:13 AM.


MobileRead.com is a privately owned, operated and funded community.