Multi-directory search

[daudi]

This is a spin-off from something else I'm playing with. The built-in iliad search will only search the current directory, and if I'm not mistaken will only search the title field, not the description.

This little script will allow you to search multiple directories and will grep the manifest file. This means you can put key words in the description field and it will match on those.

There is another script (contentlister entry) for deleting the results. Don't use the built-in delete because this deletes the original files rather than the symlinks. I think thomas posted a better way to do symlinks and I'll try using that later.

You enter the search phrase by changing the FIRST LINE of the description field of the search entry. At the moment this is "wibble", because that is what I was searching for when testing this. Use the label function to change the description, then click on the search entry.

So, in summary, these are the steps:

1. get shell access if you do not already have it
2. unzip search.zip somewhere, e.g. your PC
3. copy the whole thing to somewhere on your iliad (anywhere will do)
4. navigate to the search folder using the contentlister
5. use the config entry to select where to search
6. change the description field to your search expression
7. click on the "search" entry in the contentlister
8. when the search is complete you will see a "results" folder
9. view your results in the results folder
10. delete the results using the "delete" entry in the contentlister

I started doing this because I think I will soon have a way of tagging and taking snippets from PDFs and plan to store the tags in a tags file in the appropriate PDF container directory and snippets in a snippets file. I want to be able to search for snippets or tags.

The main limitations at the moment are:

1. no way to search within search results
2. you have to navigate to the search folder, you can't just get there quickly from anywhere
3. inelegant way of getting user input

You can set the place to start searches to any location you like. Just add entries to the file config/search.rc

I've tested this a little on my USB key with about 800Mb of files and it did not fall over. Feedback welcome.

Edit: just made a couple of updates to zip file.
Edit: the limit on the number of results is not working. I'll have to look into this.
Edit: new version---quoted search expression to allow spaces
Edit (2008-05-01): new version that I think fixes the spaces in filename problem and adds a path config tool. Also fixed hit limit.
Edit (2008-05-01): Added snippet search tool
Edit (2008-08-11): Fixed CF search problem

[yokos]

It seems like spaces are not allowed in directory names.

Search string: Krause
grep: /mnt/cf/pdf/Foundations: No such file or directory
grep: of: No such file or directory
grep: GTK+: No such file or directory
grep: Development/manifest.xml: No such file or directory
[...]
./run.sh: 87: [[: not found

After changing dir's name from "Foundations of GTK+ Development" to "Foundations_of_GTK+_Development" it is working.


[EDIT] I had got a space bug in my jukebox mpd scripts, too.

[daudi]

Thanks! I've now quoted the grep so hopefully fixed the space bug.

[yokos]

Originally Posted by daudi

Thanks! I've now quoted the grep so hopefully fixed the space bug.

Not yet. It doesn't work with spaces.
[very OT] Battery of my iLiad is low & I don't have a power adapter next to mine.

[daudi]

Originally Posted by yokos

Not yet. It doesn't work with spaces.
[very OT] Battery of my iLiad is low & I don't have a power adapter next to mine.

OK, I've looked into this a little more. To handle this nicely we need a newer version of busybox. The version on the iliad does not seem to support -print0 for find or -0 for xargs, but the current release does. This current release is version 1.10.1. Adam has released version 1.7.2 and I am not sure if that supports these options or not. I'll try to test it this evening. The alternative would be to use temporary files in the script instead. That would be a less elegant solution though.

[yokos]

Originally Posted by daudi

OK, I've looked into this a little more. To handle this nicely we need a newer version of busybox. The version on the iliad does not seem to support -print0 for find or -0 for xargs, but the current release does. This current release is version 1.10.1. Adam has released version 1.7.2 and I am not sure if that supports these options or not. I'll try to test it this evening. The alternative would be to use temporary files in the script instead. That would be a less elegant solution though.

Oh, interesting.
It should be no deal to update busybox. Termial apps are dream targets to port compared to GUI apps with all the odds.
I think Adam hasn't enabled all features of busybox during configure to make the binary small.

EDIT:

Originally Posted by README of busybox

Using busybox:

BusyBox is extremely configurable. This allows you to include only the
components and options you need, thereby reducing binary size. Run 'make
config' or 'make menuconfig' to select the functionality that you wish to
enable. (See 'make help' for more commands.)

[daudi]

Ahaaa. Very helpful, thanks. I'm not going to be able to look at this again for a few days, but this will save me time mucking around trying to figure out things that might not need figuring out. Thanks.

[daudi]

It turned out to be rather easy to make this work with the standard busybox so I have decided to do that rather than require people to upgrade their version of busybox. I have tested it and think it is working now with directories with spaces in the path and would be grateful it you could try it again Yokos.

I have also changed the way that search paths are configured. Instead of having to hard code the path in the script, I have now added a config contentlister entry that reads a file called search.rc. This file consists of one line per directory, a semi-colon and a description. The default contents on this file are:

/mnt/usb;USB
/mnt/cf;Compact flash
/mnt/free;All internal memories

You can easily add others to particular directories. With this you just click on the entry in the contentlister to cycle through the paths to search. The description field in shown in the contentlister is updated with the description of the current search path.

I've also fixed the number of hits limit code. This is currently set to 40 results. You can change this near the top of the script.

I'm updating the post at the top of this thread with this version.

[Dabon]

Dear Daudi,

Good day!
Thank you for working so diligently on such great and useful tool.
I have tried to install this tool several times, but unsuccessfully. i am not getting the message " installation successful". I am able to switch the locations for research between internal memory, USB and CF, but the launching of research is unsuccessful, the iliad does not react when I click on search folder...
Any ideas or suggestions on what I could do to get it working?

Thanks a million in advance!

D.

[daudi]

Originally Posted by Dabon

Dear Daudi,

Good day!
Thank you for working so diligently on such great and useful tool.
I have tried to install this tool several times, but unsuccessfully. i am not getting the message " installation successful". I am able to switch the locations for research between internal memory, USB and CF, but the launching of research is unsuccessful, the iliad does not react when I click on search folder...
Any ideas or suggestions on what I could do to get it working?

Thanks a million in advance!

D.

There's nothing to install so there's no need for an "installation successful" message
I suspect it probably is running but using the default search pattern. I think I probably need to re-write the first post or write another one to make things clearer.

A key thing from my first post is:

You enter the search phrase by changing the FIRST LINE of the description field of the search entry. At the moment this is "albert", because that is what I was searching for when testing this. Use the label function to change the description, then click on the search entry.

Have you used this to enter the search phrase? It is a kludge, I know, but seemed to me to be the simplest way of getting simple user input. Once you have entered your search phrase clicking on the search entry should run the search.

WARNING! At the moment the general search searches the entire manifest file. This means that if you are searching for "The Great Escape", for example, do not enter "esc" as the search phrase. "esc" will match "escape" okay, but it will also match "Description" which exists in (almost?) every manifest file.

BTW, I should also explain that the snippet-search relates to this thing.

If you have entered your search phrase correctly and it is still not working we'll have to delve a little deeper. Let me know how you get on.

[engunneer]

Originally Posted by daudi

If you have entered your search phrase correctly and it is still not working we'll have to delve a little deeper. Let me know how you get on.

I am having the same problem as Dabon.

The search folder is loaded in Newspapers/Programs/Search.
Newspapers is on the CF card (according to iLiad Settings/Archive Locations page)
The document I am searching for has a two line description. The only thing that may be odd is that the iLiad did not generate the manifest.xml. I am using another program that I wrote to generate it. It is located in documents/publications/[folder name]/ (also on the CF card)
I am editing the first description line of the search program using the tagging tool. In today's case, the search term is banding. The config program is switching modes just fine, but the search tool runs and no change is shown in contentlister (the screen does refresh). Is there some error log I can look at?

Thanks,

[daudi]

Hi engunneer,

There's no logging at the moment, but I'll look into it. For now, some questions that might help to track down the source of the problem:

1. When you click on the search entry (after setting the description to "banding"), does it seem to do anything at all? Do you get the two moving bars at the bottom of the screen that show that the iliad is thinking about something? (It did this for about 2 mins while searching 1Gb of files just now)
   
2. Do you get a "results" contentlister entry? If so, your results are in there.
   
3. What are the contents your config/search.rc? What is your currently selected config?
   
4. You could try commenting out line 86

   rm match.lst match.tmp

   and see if those files (match.lst and match.tmp) are created.

[engunneer]

Thanks for getting back to me.

1. The LED blinks and the White Loading bricks at the bottom do show for about 2 seconds. My CF card is 2GB and about half full. None of the bricks turn black.
2. No results at all (I do have the entry to delete results)
3. I am using the default one you have posted above. I have tried all of the settings, but Compact flash is the one that should work in this case.
4. the files are created, but both are empty.

If I can get this working, then I will try to get your snippet tool working (on windows), which looks very impressive. Then I will publish the project for the iLiad that I have been working on.

Update: Setting to "All internal memories" does at least run long enough to get one black box to fill.

At this point, I am suspecting that the manifet.xml of my target files are somehow wrong? I can try to upload an example doc tomorrow.

[daudi]

Originally Posted by engunneer

4. the files are created, but both are empty.

[...]

At this point, I am suspecting that the manifet.xml of my target files are somehow wrong? I can try to upload an example doc tomorrow.

match.tmp is a list of manifest files to check, so if that is empty it suggests that it is not even finding any manifest files. One thing that springs to mind now that you mention windows is case sensitivity. manifest.xml != manifest.XML and I think I have seen some windows programs cause the extension to be uppercase. The find command that I use only searches manifest.xml files (lower case extension).

find "$STARTDIR" -type f -name 'manifest.xml' > match.tmp

That would explain why match.tmp is empty. There's not actually much that can go wrong in this command: either the STARTDIR (starting directory) is wrong, or there are no files named 'manifest.xml'. Can you verify the case of the extension?

[engunneer]

I can verify the extension. Extensions in uppercase bug the heck out of me, and I also wrote the program that is generating them.

I also have some documents that were nout generated by my program, and I have manually tagged them on the iLiad, and the search program still doesn't find them. I have not tried to experiment with ssh access, but theoretically could if you think it will help me diagnose this. Thanks for taking the time to try and help.

Is there something I can hardcode instead of $STARTDIR, so that it is known good? should I try running it via mrxvt?

Thanks.