Indexeses

[NetSlut]

Mmm, I think Calibre (a wonderful program that I've been using) does something different: it seems to me to be more focussed on device management and conversion of ebooks, rather than a pure indexing tool.

I think what I want is something more like the Apple iPod, doing one thing exceptionally well.

All of the features wished for so far shouldn't prove to be much of a problem, although I'm hesitant over the .txt file support since I think so many text files aren't ebooks that they will need special handling.
But everything sounds feasible, so as I say, let's give it a few days for ideas to percolate and then we'll take stock and see what can be done.

[kovidgoyal]

Originally Posted by NetSlut

Mmm, I think Calibre (a wonderful program that I've been using) does something different: it seems to me to be more focussed on device management and conversion of ebooks, rather than a pure indexing tool.

I think what I want is something more like the Apple iPod, doing one thing exceptionally well.

All of the features wished for so far shouldn't prove to be much of a problem, although I'm hesitant over the .txt file support since I think so many text files aren't ebooks that they will need special handling.
But everything sounds feasible, so as I say, let's give it a few days for ideas to percolate and then we'll take stock and see what can be done.

The calibre GUI was written originally to index books. What about indexing does it not do well?

[NetSlut]

Originally Posted by kovidgoyal

The calibre GUI was written originally to index books. What about indexing does it not do well?

Well, there are a number of things that I think would be done better if the program didn't have to pay attention to the needs of the conversion and device management; not enough attention is paid to indexing by format, for example, and personally I want to have an index that will actually tell me the full path and file name to each of my ebooks, or to gather them into a single place, much like iTunes does.

As I said before, Calibre is a very useful program that I will continue to use (even though it doesn't work properly on my mac, and tends to corrupt the prs-505 sometimes), but this is a different program with a different focus.

[kovidgoyal]

Originally Posted by NetSlut

Well, there are a number of things that I think would be done better if the program didn't have to pay attention to the needs of the conversion and device management; not enough attention is paid to indexing by format, for example, and personally I want to have an index that will actually tell me the full path and file name to each of my ebooks, or to gather them into a single place, much like iTunes does.

As I said before, Calibre is a very useful program that I will continue to use (even though it doesn't work properly on my mac, and tends to corrupt the prs-505 sometimes), but this is a different program with a different focus.

Feel free to write you own program, I'm just curious as to what features you think are missing.

calibre does gather all your books into a single place (a database in versions <= 0.4.83 and a user specified folder in higher versions). Not sure what you mean by indexing by format. You can filter the list of books by format, what else do you want to do?

[pilotbob]

Originally Posted by NetSlut

Mmm, I think Calibre (a wonderful program that I've been using) does something different: it seems to me to be more focussed on device management and conversion of ebooks, rather than a pure indexing tool.

You can add tags to books and search them by tag. You can sort the books by Title, Author, Publisher, Series. Book info does not show you the file name like iTunes does, but the newest version (still in beta) organizes the files on your hard drive by Author, Title so they are very easy to find if you want to transfer ebooks to a device using the OS Explorer/Finder.

Yes, I think it does need a bit more work. I would like to see a tag list / cloud so you can navigate through them without having to sort on them. Also, I think the list has to be paged rather than just listing all the books in the grid since I have seen reports of large (10k) collections really slowing down the UI.

You should download the beta to your Mac... it has come along way. Not sure what version you are using. Then, you can add features you like to it rather than starting from scratch...

In addition to the tag list / cloud navigation it would be nice to Like, it would be nice to display the file path somewhere. This is actually a new feature in the beta where the files are stored in the file system rather than the database.

BOb

[kovidgoyal]

Originally Posted by pilotbob

Yes, I think it does need a bit more work. I would like to see a tag list / cloud so you can navigate through them without having to sort on them. Also, I think the list has to be paged rather than just listing all the books in the grid since I have seen reports of large (10k) collections really slowing down the UI.
BOb

Actually it's the sort/filter operations that become slow, no the UI. Paging wont help that, since for the results to be correct, a sort/filter has to run through the full database.

[pilotbob]

Originally Posted by kovidgoyal

Actually it's the sort/filter operations that become slow, no the UI. Paging wont help that, since for the results to be correct, a sort/filter has to run through the full database.

Ah... what db are you using? It seems slow in that case, cause 10k records isn't alot. Do you have the data indexed by title/author/tags so that the db doesn't have to do a table scan and it can do an index scan?

Do you create tag index tables or just sort through a tags field in every record to find stuff?

BOb

[nekokami]

In a lot of my books currently, the metadata (title, author) is included in the file path. I'd like a utility to be able to be configured to parse this. How people have organized books tends to be individual, so it would need to be configurable. In my case, I have all my books in a "books" directory, and within that, I have directories named after each author, in Last, First format. Within those directories, things vary a bit more, but usually there is a level of format directories, and then within that, filenames may include the author's name as well as the name of the book.

So in my case, I'd like to be able to configure the indexing utility to check within a given directory and assume that the next directory level down will provide the author's name, then search recursively within those directories for all files and remove the author's name from the filename to get the book title, if appropriate.

Once the author's name and book title are available, of course, it would be great to be able to get other info from an online database.

One other feature I'd like is the ability to enter or import my paper book collection, and tie my paper inventory to my electronic inventory. I'm gradually trying to replace all my paper books, especially fiction, with ebook versions. I have Readerware and a barcode scanner, so I plan to have a full online inventory of my paper books "soon." Readerware has some nice lookup features that might be able to provide me with an estimate of the resale value of my paper books, so it might also be useful to have a system be able to look up and store the lowest cost available (of a specified format or formats) to purchase an ebook replacement. I could then calculate how much it would cost, net, to sell my paper books and buy commercial versions of ebook replacements (when available).

[kovidgoyal]

Originally Posted by pilotbob

Ah... what db are you using? It seems slow in that case, cause 10k records isn't alot. Do you have the data indexed by title/author/tags so that the db doesn't have to do a table scan and it can do an index scan?

Do you create tag index tables or just sort through a tags field in every record to find stuff?

BOb

Sqlite, and it is fully indexed on all sort fields. I haven't really looked into what is causing the slowdown, primarily because this isn't a problem for most users, but I suspect some flaw in the database design. I'm just not enough of a sql guru to be able to figure it out easily.

[pilotbob]

Originally Posted by nekokami

In my case, I have all my books in a "books" directory, and within that, I have directories named after each author, in Last, First format. Within those directories, things vary a bit more, but usually there is a level of format directories, and then within that, filenames may include the author's name as well as the name of the book.

That is very similar to how the calibre beta organizes files. You specify the top level folder. In that folder is a folder for each author. The folder is named based on the authors name in the calibre database. So, you can put the author as lastname firstname if you want. Within each author folder is a folder for the book title. With that are the book files and a .obd file an sometimes a cover picture graphic file. While it would be nice to be able to specify this the current layout works perfectly well for me and it is easy to find an ebook file that I want to transfer to my device.

Originally Posted by nekokami

Once the author's name and book title are available, of course, it would be great to be able to get other info from an online database.

It does do that. Generaly getting the ISBN and Summary info from isbndb.com.

Originally Posted by nekokami

One other feature I'd like is the ability to enter or import my paper book collection, and tie my paper inventory to my electronic inventory. I'm gradually trying to replace all my paper books, especially fiction, with ebook versions. I have Readerware and a barcode scanner, so I plan to have a full online inventory of my paper books "soon."

Hmm... I don't use readerware... I use librarything for that. Although I have choosen to enter "books I've read" in library thing rather than use it to index files/books that I have/own. Of course you could use it that way via tags. Then I use calibre to "index" my ebook files and also do conversions to LRF if needed. I don conversions to Mobi using other command line tools then add those files to calibre. But, I think there are future plans to be able to plug in other command line tools.

Hey, there is always room for more software in this world. But, since calibre is open and Kovid accepts patches it seems to make sense to add some of these features to it rather than starting with a clean slate.

BOb

[Elsi]

I have yet to download and use the beta version of Calibre, but it's on my list of things that I really want to do. So, I'm not sure how Calibre manages the inventory of books in this release.

What *I* want is a program that will create an index of the books -- exactly where they are in the file system. I do not want a program that moves them into a new location on the hard drive(s). I do not want them filed in folders by author since I have chosen to organize them by source. But an index would allow me to view by author *across the existing directories*. And that's a very good thing.

[NetSlut]

I think the flexibility of understanding and optionally organising directory paths is a must for what I wanted to see. I have a nasty habit of forgetting certain author's names and just remembering the book titles, especially if they are one-off's, so I'd want to be able to have different paths set up but remain in one index.

I want something that would show my collection in ways that are meaningful to me at the time of searching, so I can see which books are missing a ebk format, or which Stainless Steel Rat books have yet to come out.

Maybe part of my problem in missing these things is that I haven't tried the beta version of Calibre :-) and I haven't had much time with the release version either (it's the latest release, by the way); but I think shoehorning what I want to do into Calibre, would be more difficult.

[kovidgoyal]

Originally Posted by NetSlut

I want something that would show my collection in ways that are meaningful to me at the time of searching, so I can see which books are missing a ebk format, or which Stainless Steel Rat books have yet to come out.

In the search bar enter:

format!ebk

for all books that dont have ebk format. Assuming you've tagged the books that have yet to be released:

series:"Stainless Steel Rat" tag:unreleased

will do the trick. I doubt there's any collection querying feature that you can think of that calibre can't do.

And to address the concerns about folder organization. My perspective is: does it really matter? Why do you organize your books into folders in the first place? I'm guessing, so that you can find a particular book faster and keep the various files belonging to a "book" in one place. I claim that having the books managed by calibre allows you to find them much faster than in any conceivable folder layout. Not only that, you get automatic support for rich metadata (covers, summaries, series, ratings, etc) and the ability to seamlessly convert either individually or in batch any given subset of your books at the click of a button.

[nekokami]

Originally Posted by pilotbob

Hmm... I don't use readerware... I use librarything for that. Although I have choosen to enter "books I've read" in library thing rather than use it to index files/books that I have/own. Of course you could use it that way via tags. Then I use calibre to "index" my ebook files and also do conversions to LRF if needed. I don conversions to Mobi using other command line tools then add those files to calibre. But, I think there are future plans to be able to plug in other command line tools.

Hey, there is always room for more software in this world. But, since calibre is open and Kovid accepts patches it seems to make sense to add some of these features to it rather than starting with a clean slate.

BOb

Integration with Librarything would be fine, too (or instead). I have a lifetime account there (I found Librarything after I already had gotten Readerware). The point is to be able to manage both my ebook and pbook collections within one system and see how far out of sync they are. I have quite a few "format shifted" ebooks, and I want to make sure my ebook collection is accounted for in either paid-for commercial ebooks or matching pbooks. Also, as previously mentioned, I'm trying to duplicate/replace my paper library entirely with ebooks, as nearly all my pbooks are currently in storage due to lack of shelf space, and I'm going through withdrawal symptoms on some of them.

BTW, does Calibre consider CBZ/CBR as supported filetypes? Due to space considerations, I've been format-shifting my manga collection, as well. Sometimes I convert the CBZ files to PDF using various utilities, but not always.

[kovidgoyal]

Originally Posted by nekokami

BTW, does Calibre consider CBZ/CBR as supported filetypes? Due to space considerations, I've been format-shifting my manga collection, as well. Sometimes I convert the CBZ files to PDF using various utilities, but not always.

Yes it does.