Am I Missing Something About Search?

[Gary_M_Mugford]

Hi all,

Does Calibre have the facility to access complex searches from disk files (editing in a full-screen editor is much, much easier than in a edit box line-editor)? Is there a way to suck in these disk files to a saved search name? Say for example, I would like saved search ForTAidcode123 to go out to a designated area and suck in the search from ForTAidcode123.txt and then run it. This obviously would allow for external editing of the search code.

I've been through the forum search here and I haven't chanced upon the requisite ability being described. What I HAVE been doing is a bit esoteric and demands more of my ad hoc solution than I would like. And all of this for a Bridge buddy of mine who's also a college prof with three Teaching Assistants operating as his minions.

I've set up Calibre on their system so that students can submit essays to a communal folder that is sucked up into Calibre. Each TA has a set of students he or she's supposed to monitor new submissions from and read them. It was easy at first, when the system was a lot smaller despite being low-tech. The TA could sit down, type in a search for each student one by one (the submissions being in reverse date order) and see what was needed for reading. Well, more students this year and two more TA's meant the system started to break down.

So, I created a text file for each TA with a list of the students in the form of a formatted list, each line after the header having a single student's name in the form of: 'or authors:"=Student One (or whatever)".' The text file can be copied and pasted into the search with the old Ctrl-A, Ctrl-C, Ctrl-V sequence (this is obviously Windows). And I use the search buttons to save the search and the Tag Browser to edit it's name to ForTAidcode123 (I'm consistent, if not original). It works, but it DOES basically demand somebody remember to do the saving/renaming sequence when the original file is edited. Murphy feels imminent. I also think I will be coming acropper of the limit for either clipboard text contents OR the length of text in the search box before implosion.

I looked at tagging, but I couldn't see a way to make it work with the import routines. Same issue, I need a list comparison of variable data. And being able to edit the list(s) outside of Calibre would make easy for the minions to work and for the prof to play.

Still, given all the nice SQL tools and stuff already in Calibre and the dandy new DB backend, it seems to me I must be missing something.

Or is this all so micro niche that it doesn't merit further investigation?

[chaley]

Have you looked at creating a user category for each TA, then populating that category with the students in that TA's section? This scheme would eliminate the external file. The user category would show the students in alpha order, making it as easy to see the list of students there as in the external file. The search is easy: click on the user category.

To make it even easier, create an "enrollment record" book for each student. Tag that record with the section(s) that the student is in. Given that, creating or updating the user category is as simple as searching the enrollment records for the section name, selecting all the books, then dragging the author (the student's name) to the user category.

[Adoby]

Calibre also has a command line interface and using that it is possible to use the functions of calibre in a custom way. But you have to write the scripts and the supporting apps yourself, or pay someone to do it. Or give someone academic credits for doing it...

Check out: http://manual.calibre-ebook.com/cli/cli-index.html

A very simplistic way to do it could be like this:

You need an app for students to use when they submit their texts. Could be a web app run from a server. When the text is submitted the student has to add his/her name as author and the name of the assignment and name/email of the TA, possibly by choosing from a list.

When the text is submitted an opf-file is also created that stores all the metadata entered in the app. Author, email to author, timestamp, original filename, assignment, comments, checksum, TA and so on. If the text already is in a ebook format, then the fields are populated from metadata embedded in the file.

The text and the opf-file are sent, by the app, to a central server using mail, ftpor httpor whatever you prefer. Could perhaps also allow submission using CD or thumbdrive.

On the central server, the text is (re)converted to a ebook format (preferably epub) and the metadata is set from the opf-file. If successful a confirmation mail is sent to the author, along with a copy of the newly converted book.

The server then use commandline tools from calibre to add the text to a central calibre library, using the opf-file to supply the metadata.

Also the text is automatically mailed to the correct TA, or stored in a folder, with the metadata set directly in the file.

The TA adds the book to a local calibre install. The TA updates the comments field as necessary and possibly also other fields, possibly even edits the book using Sigil to insert inline comments and suggestions, and sends the book back to the central server using the web app.

The central server adds the new copy of the book to the central calibre library, using the embedded metadata, and again sends copies to the author and the TA. The text of the mail is taken from the comments field. There are now at least two copies of the book in the library, the submission and a copy with comments from the TA. Possibly also several previous versions along with copies with previous comments from the TA.

Except for adding submissions from the author and commented copies, the central calibre library can be made read-only accessible using calibre server.

It is possible to list submissions for example by author, date, assignment, TA and so on.

The professor and other staff can also download submissions from the calibre server and add comments as well. When the book is submitted back, using the Web app, new copies are mailed to both the author and the TA.

[Gary_M_Mugford]

Originally Posted by chaley

Have you looked at creating a user category for each TA, then populating that category with the students in that TA's section? This scheme would eliminate the external file. The user category would show the students in alpha order, making it as easy to see the list of students there as in the external file. The search is easy: click on the user category.

To make it even easier, create an "enrollment record" book for each student. Tag that record with the section(s) that the student is in. Given that, creating or updating the user category is as simple as searching the enrollment records for the section name, selecting all the books, then dragging the author (the student's name) to the user category.

Chaley,

I worked through a tagging system a couple of years back. But changes in calibre (accepting a folder, to read in automatically) and the way my friend worked his system between the now three campuses, made me rethink things. And I came up with the solution I outlined waaaaay above. And it really does work. I'm just worried it will stop working due to limitations in Windows more than anything else.

I should further explain that the students are all taking the professor's course, at whatever year level, and this system if merely for submitting papers and essays. The discreet files are copied into Calibre for later reading on the TA's iPads, who mark and comment and send the summary to my friend. The files are also referenced for later submittal to a plagerization service. (It's a catalog export to a csv file and then I've written a small app that creates a text file with the file locations of the original DOCx's for mass uploading to the testing site). All of this later stuff is based on date added to the Calibre database. I know, I know, kludgy as all hell and it would't be needed if the college had a decent centralized content management system, but it doesn't.

It's obvious that there is no direct disk input function within Calibre's GUI to get this done. Sigh. And my needs are so esoteric, it would be a waste of time to have Kovid look at it, shake his head and send a NO reply.

So, I will persist with that that is working and pray my luck (in these sort of things) holds out. One TA is monitoring 100 students currently, which makes me think that's the thin edge of the maximum. But a total of 300 is juuuuuust about at the edge of what the professor can handle in any given term. I'm going to rename old searches to include a year code as they go from one year to the next.

As it is, I will be alerting each TA that they are in charge of providing me emails, copied to their boss, of their master list with any new edits. Then, it's up to them. And all I have to do to maintain the house of cards is to spend all of three minutes copying, pasting and renaming in Calibre. Am I REALLY that lazy that I can't afford three minutes every now and then? Well, speaking as a programmer, yeah! But in reality, no.

Thanks for suggestions to a tag system that would, probably equally as well as what I've got.

Regards, GM

[Gary_M_Mugford]

Originally Posted by Adoby

Calibre also has a command line interface and using that it is possible to use the functions of calibre in a custom way. But you have to write the scripts and the supporting apps yourself, or pay someone to do it. Or give someone academic credits for doing it...

Check out: http://manual.calibre-ebook.com/cli/cli-index.html

A very simplistic way to do it could be like this:

Adoby,

A very complete roadmap of a solution. Unfortunately I'm a little out of my depth here with the CLI. It's one of the reasons of late (the other being some health issues including recently concluded cataract surgery on both eyes), that comments here on the forum have been far and few inbetween. There's also the fact that Calibre does everything I need, just not everything OTHER PEOPLE need.

My ad hoc system works. I have to be involved, but it works. And I honestly can spare the few minutes every now and then that it requires to work in 'manual' mode. My only fear is that my way of doing things will break under the limitations of the OS (Windows). Although, being the laziest man alive, I would like to out from under my onerous workload [G]

Between your answer and Chaley's, if I ever decided to get truly serious about a solution, I now have a starting place. Or I can pass it along and let the young'uns do their worst to solve the workflow of submitting their pearls of wisdom to the ascerbic old man who teaches them.

Thanks for taking the time to flesh out and idea. Appreciate the time you put into it.

Regards, GM

[chaley]

Originally Posted by Gary_M_Mugford

Chaley,

I worked through a tagging system a couple of years back. But changes in calibre (accepting a folder, to read in automatically) and the way my friend worked his system between the now three campuses, made me rethink things. And I came up with the solution I outlined waaaaay above. And it really does work. I'm just worried it will stop working due to limitations in Windows more than anything else.

Gary, you sound like you have things well under control. That said, your answer implies that I didn't explain what I said well enough, so even though this message is probably superfluous ...

A user category is not a tag. Instead it is a collection of metadata items such as authors, series, publishers, or tags, providing a level of indirection. For example, you could create a user category called "Favorite authors" and populate it with authors you like. Same with series. They appear in the tag browser, can be opened to see their contents (like other categories), edited, and searched. You can drag n drop metadata items onto the category to add them. When you search using a user category, it (in effect) does a search for each item in the category and 'or's the results together.

The advantage of the user category is that the TA can easily see the list of names in the category by opening the category. This is harder when using saved searches. Not a lot harder, though. However, since your are asking the TAs to send you emails, clearly they know who is in their sections. Instead of sending emails they could drag the student's name into 'their' user category.

Another thing to mention: the calibredb command can create a saved search. You might be able to maintain the saved searches using this command, but it might not be worth fighting with the shell to get the arguments quoted correctly.

[Gary_M_Mugford]

Originally Posted by chaley

Gary, you sound like you have things well under control. That said, your answer implies that I didn't explain what I said well enough, so even though this message is probably superfluous ...

A user category is not a tag. Instead it is a collection of metadata items such as authors, series, publishers, or tags, providing a level of indirection. For example, you could create a user category called "Favorite authors" and populate it with authors you like. Same with series. They appear in the tag browser, can be opened to see their contents (like other categories), edited, and searched. You can drag n drop metadata items onto the category to add them. When you search using a user category, it (in effect) does a search for each item in the category and 'or's the results together.

The advantage of the user category is that the TA can easily see the list of names in the category by opening the category. This is harder when using saved searches. Not a lot harder, though. However, since your are asking the TAs to send you emails, clearly they know who is in their sections. Instead of sending emails they could drag the student's name into 'their' user category.

Another thing to mention: the calibredb command can create a saved search. You might be able to maintain the saved searches using this command, but it might not be worth fighting with the shell to get the arguments quoted correctly.

Chaley,

I took a brief look and I understand the opportunities you suggested looking at better. The positives are the aftermath of the work creating it when it comes to doing searches. Near as I can tell, when I drop down the search box to look at the history, the user category searches only show the @_Gary:true line rather than the FULL SPACE-CONSUMING LIST that doing the named searches do. (An aside to Kovid, shouldn't a saved search only show the search:"=SAMPLE_BIG_SEARCH_WITH_LOTS_OF_NAMES" rather than the expanded out list of ALL the names that were in the big compound search when looking at the LIFO history of searches?). And once you got a big list into your user category, adding a new one is pretty straight forward and easy enough to do. Spelling errors are close to non-existent. There's also the second level of filtering so that I could create a user category of my favourite authors' mysteries, although that level of granularity is more theoretically good than I have use for. The entry in the tag browser also offers you the all encompassing look, or, when expanded, each individual author.

The downside, no easy importation of a list that I could find. I started one for my own favourite authors in my own calibre and it was ... non-trivial to add the names. After a delayed startup which I understand the reasons for, I had to scroll through available items to add to applied items by double-clicking on the author. No paste function from an externally copied list. If I was to have ten favourite authors, this would not be a downside. But I have dozens. It just seems faster to open a text file, copy and then paste the names into the search box and hit enter. Even changing or adding one name, it's still the same process ... assuming Windows co-operates. I don't care where, who or whether it was simply a spelling correction, the mass copy and paste as an update for compound searches works well (within the limits of my testing) Also, I don't keep the tag browser open much. Somewhere in the threads that talked about adding files being faster if the browser wasn't open, I decided I liked faster better and I prefered not 'wasting' the screen real estate. So for _Gary to have its own entry, rather than as an entry underneath User Categories is a negative.

Am I going to work to finally create a useable _Gary for myself with all the authors I want to have at a click of a button (or two)? Yes. I think so. I will also create a _GaryRecommends with sub-filtering by SFincluded because so few of my friends want SF recommendations slotted into their mystery list to try out. I think User Categories have a use. But in the kludge I've thrown together, I think I will stick with what's working till it ain't.

Thanks for pointing me towards the feature and then clarifying things when I didn't get it the first time.

Regards, GM