Learner questions

[mike_bike_kite]

I'm getting more into Calibre and the more impressed I'm becoming but I have a few questions:

Some of my books are in the format "author name - series name - book title.type" and this causes issues. Is there anyway to get Calibre to understand this type of format or should I rename these files? Some books ended up with the title and author swapped over - is there a way to swap them back without editing the data manually?

If I add meta info and covers to a whole folder of books I might get problems with 10% (still pretty good though). Is there an option to see which ebooks don't have covers yet?

[Starson17]

Originally Posted by mike_bike_kite

Some of my books are in the format "author name - series name - book title.type" and this causes issues. Is there anyway to get Calibre to understand this type of format or should I rename these files?

Yes Calibre will understand it. No, don't change filenames. Change the regular expression in Preferences |Add/Save and turn on Get metadata only from filename. I'd write the regex for you, but it's easier if you give a sample filename (do your series have a number in them?)

Some books ended up with the title and author swapped over - is there a way to swap them back without editing the data manually?

Yes (I learned a few days ago). Edit the metadata (select it and press "E") then hit the button to the right of "Title".

. Is there an option to see which ebooks don't have covers yet?

Yes. In Search box type: cover:false

[mike_bike_kite]

Most file names are like this:

Arthur C Clarke - Patent Pending.txt

But a small percentage might be like this:

Arthur C Clarke - Rama 01 - Rendezvous With Rama.lit

I found the regular expression it currently has (?P<title>.+) - (?P<author>[^_]+) but I couldn't see how you could suggest multiple expressions ie one expression for the first type of file name and another for the other type of file name. I'm familiar with traditional regexp but these are different - I assume the <title> refers to an internal variable for the title but the ?P etc seemed odd.

Yes. In Search box type: cover:false

That worked though it's not very integrative. It's a shame they don't have a simple display column for cover and then you could just sort by this field to find those without covers.

PS I'm using the save option to save things but I have no real idea what I'm saving - is it the new meta data I'm creating?

[Starson17]

Originally Posted by mike_bike_kite

Most file names are like this:

Arthur C Clarke - Patent Pending.txt

But a small percentage might be like this:

Arthur C Clarke - Rama 01 - Rendezvous With Rama.lit

I found the regular expression it currently has (?P<title>.+) - (?P<author>[^_]+) but I couldn't see how you could suggest multiple expressions ie one expression for the first type of file name and another for the other type of file name.

Use this for your regex:

^((?P<author>([^\_0-9]+)(?=\s*-\s*)(?!\s*-\s*[0-9.]+)|\b))(\s*-\s*)?((?P<series>[^0-9\-]+) ([-#] ?)?(?P<series_index>[0-9.]+)?\s*-\s*)?(?P<title>.+)

It handles both your examples and many more.

I'm familiar with traditional regexp but these are different - I assume the <title> refers to an internal variable for the title but the ?P etc seemed odd.

That's part of the Python format string

That worked though it's not very integrative. It's a shame they don't have a simple display column for cover and then you could just sort by this field to find those without covers.

I prefer having the cover:false option. It can be combined with other searches to e.g. show me all books that don't have a cover or don't have a summary. (Have you found the cover browser?)

PS I'm using the save option to save things but I have no real idea what I'm saving - is it the new meta data I'm creating?

Metadata is saved automatically. Are you referring to Save to Disk? That's used to save your book to the disk. I'm not sure what other "save" you may be referring to.

[mike_bike_kite]

All the above worked well but I still notice a few odd things though:

• If a file name containing the book title is missing an apostrophe then the book isn't found when looking for meta data - this should be corrected as most folks will remove apostrophes from file names.
  
• If a file is called "Tale of two cities.txt" then the book won't be found. I can understand this with lesser titles but surely it doesn't need an author name to work out which book this is?
  
• Many old classics will appear in the database under lots of different guises ie the original work, the book with commentary, perhaps a graphic version and then perhaps a study version. Rather than choosing none would it be better to choose the original version and grab the meta data and cover for that.
  
• Does Calibre keep the original folder the file was in as meta data anywhere - I'm not sure how to distinguish the books in my science folder with those in my science fiction folder. I wasn't sure but is it put into the tags?
  
• If I enter the meta data and cover for "Tale of two cities.txt" then it would be nice if it automatically updated the meta data for "Tale of two cities.lit".

Calibre's great but periodically it might be worth grabbing someone's library and seeing how easy it is to add all the useful info.

[itimpi]

The point is that the file extension should not be on the Title. Then Calibre would store both formats in the same folder under a single entry in Calibre.

The easiest thing tends to be to correct one of them to the correct value and then use the Calibre Merge facility to combine them under a single book entry.

[Starson17]

Originally Posted by mike_bike_kite

If a file name containing the book title is missing an apostrophe then the book isn't found when looking for meta data - this should be corrected as most folks will remove apostrophes from file names.

If a file is called "Tale of two cities.txt" then the book won't be found. I can understand this with lesser titles but surely it doesn't need an author name to work out which book this is?

You're asking for fuzzy matching to be implemented in the metadata fetch. It has been implemented in various parts of Calibre. I wrote fuzzy matching into the "If books with similar titles and authors found, merge the new files automatically" option. It's the "similar" in that option. I believe Charles wrote it into the matching algorithm for identifying books on Devices. If you want it, post an enhancement request on the bug tracker. However, there may be some drawbacks if you fuzzy match the title during an automated metadata fetch of multiple books. Perhaps it would make sense to only use it during bulk fetching if the author is also specified.

Many old classics will appear in the database under lots of different guises ie the original work, the book with commentary, perhaps a graphic version and then perhaps a study version. Rather than choosing none would it be better to choose the original version and grab the meta data and cover for that.

I'm not sure of your point. AFAIK, Calibre uses the first match for bulk fetches, and otherwise, shows the first 20 or so matches. I'm not sure when you think it is "choosing none." I think it only does that when no matches occur.

Does Calibre keep the original folder the file was in as meta data anywhere -

No

I'm not sure how to distinguish the books in my science folder with those in my science fiction folder. I wasn't sure but is it put into the tags?

Nothing is put into tags other than what you enter, or you allow to be added during metadata fetch. Just add all the books from the SF folder and tag them.

If I enter the meta data and cover for "Tale of two cities.txt" then it would be nice if it automatically updated the meta data for "Tale of two cities.lit".

It does when you Save to Disk or Send to Device. Calibre considers this to be one book, assuming you have entered into a single record. If not, use the Merge function (another feature I wrote and am proud of, despite the help I needed from Kovid). Merging will keep the metadata and put the two books in a single record. (If they were added to separate records, you need to turn on the "If books with similar titles and authors found, merge the new files automatically" option.)

Calibre's great but periodically it might be worth grabbing someone's library and seeing how easy it is to add all the useful info.

I'm not sure what this means. I added 14K plus books, and during that period wrote a half dozen bits of code to make it easier for me, then shared them. Most of the bits were designed specifically to make adding metadata or books easier. The difficulties involved in adding new books and metadata is the one area of Calibre that I'm really familiar with.

[mike_bike_kite]

Don't get me wrong - I think Calibre is great but it seems to require a hell of a steep learning curve.

For instance seeing the correct author and title in the meta data and being told that an entry couldn't be found - it turned out an apostrophe was missing from the book title. I found the same issue if an author's name had periods after their initials when these weren't in the database. I understand why it's not matching but I can't understand why you can't strip the punctuation before trying to do a match. Sometimes there was text in brackets within the file name this had to be stripped before I could get a match. Often the book title would appear by itself with no author - the books were famous enough to not need any more detail but these books ended up with "Unknown" in the author field and no match in the database. Quite frustrating.

At other times the file might be named "book title by author" and it seems like an attempt could be made to spot these type of matches automatically. I mentioned before that sometimes the files had the author and the title mixed up but it should be possible to check this issue automatically as well.

I appreciate that this might add a few more milliseconds to a search but it added an age to me setting up my library. If you're just querying a MySQL database then I'd be happy to write a stored proc to do all the above.

PS @itimpi - the file extension mentioned was just a file extension and not a part of the name. I came across multiple versions of books all named the same but with different file extensions. The problem was that I need to search for meta data (and book covers) for each individual file which added to the time it took me to build my library.

[Starson17]

Originally Posted by mike_bike_kite

For instance seeing the correct author and title in the meta data and being told that an entry couldn't be found - it turned out an apostrophe was missing from the book title.

Your apostrophe issue is a simple overspecification of the search. Just remove all but one important word from the title, leave the author and you will find many matches. Do the same for authors - just leave the last name. It's not Calibre that needs to strip the punctuation, it's the various data sources. Removing punctuation on the Calibre side will cause more search failures, not less, because the data sources have the punctuation and the matches will fail.

To do this on the Calibre side, it would need to muck with the contents of the author and title before the search request was sent to each data source. That might help in some situations, but you wouldn't be able to see what you were searching for. If I put the correct author in, and Calibre searched for something else, I'd be annoyed.

[DoctorOhh]

Originally Posted by mike_bike_kite

I came across multiple versions of books all named the same but with different file extensions. The problem was that I need to search for meta data (and book covers) for each individual file which added to the time it took me to build my library.

If these were the same book in different formats then you could have handled it by having the If books with similar titles and authors found, merge the new files automatically feature checked under Preferences - Add/Save - Adding books tab when adding books. Then all those various formats would be in one "book" and you would just need to update the metadata once for all formats.

Or

After the fact highlight all of the same books in various formats and use the merge feature to put them all under one "book" and then update the metadata for all formats at once.

[mike_bike_kite]

Originally Posted by Starson17

Your apostrophe issue is a simple overspecification of the search. Just remove all but one important word from the title, leave the author and you will find many matches. Do the same for authors - just leave the last name. It's not Calibre that needs to strip the punctuation, it's the various data sources. Removing punctuation on the Calibre side will cause more search failures, not less, because the data sources have the punctuation and the matches will fail.

To do this on the Calibre side, it would need to muck with the contents of the author and title before the search request was sent to each data source. That might help in some situations, but you wouldn't be able to see what you were searching for. If I put the correct author in, and Calibre searched for something else, I'd be annoyed.

I assumed the database was held on the Calibre side rather than having to go to an external server so I guess there's not much you can do apart from request a better interface. You could perhaps have an intermediary database to store the name of the file, along with the final book name and author chosen by the user. Now if a file name appears you could see if it's in the database and whether more than 90% (for example) users chose this book and title then return those details. You wouldn't have to store every book in the database but just those that are often looked up - the idea is to remove as much of the manual work as possible.

This could be done just for those books which weren't matched using your current method. Alternatively it could be done as the first pass which might mean much faster retrieval of the data.

You mentioned entering a library of 14k books so even if only 5% of these books needed manually adjusting this would still mean 700 books need adjusting.

Originally Posted by dwanthny

If these were the same book in different formats then you could have handled it by having the If books with similar titles and authors found, merge the new files automatically feature checked under Preferences - Add/Save - Adding books tab when adding books. Then all those various formats would be in one "book" and you would just need to update the metadata once for all formats.

Perhaps but then you have to remember that most users haven't a clue about all these different options and nor do they want to learn some arcane programming language to use the software. I think it's great such options exist but I just can't see why a user needs to use them when the program could sort much of this out with a little fuzzy logic.

[DoctorOhh]

Originally Posted by mike_bike_kite

Perhaps but then you have to remember that most users haven't a clue about all these different options and nor do they want to learn some arcane programming language to use the software.

Thank God I don't have to use some arcane programming language to use Calibre or I would be lost.

Originally Posted by mike_bike_kite

I think it's great such options exist but I just can't see why a user needs to use them when the program could sort much of this out with a little fuzzy logic.

I love the option and the fact that it errs on the side of not making a mistake and accidentally overwriting a book using imperfect fuzzy logic.

Don't get me wrong It would be nice to have a wonderful algorithm to properly handle my mistakes or miss-edits. Right now it seems that 10% of effort has gotten us 90% improvement and the next 10% improvement in the fuzzy logic area might take a 90% effort.

I'm looking forward to your future ideas and I hope you have skills I don't to jump in and add some effort to the project.

[Starson17]

Originally Posted by mike_bike_kite

I assumed the database was held on the Calibre side rather than having to go to an external server so I guess there's not much you can do apart from request a better interface.

It's worse than that. There is no single database, there are four now, and more on the way. Google, Amazon, ISBNDB, and LibraryThing. They have different datasets, search formats, etc. The same book in one may have a slightly different title in another. Behind the scenes Calibre is getting bits of data from many different places and making it look uniform - like it all comes from a central server.

You could perhaps have an intermediary database to store the name of the file, along with the final book name and author chosen by the user. Now if a file name appears you could see if it's in the database and whether more than 90% (for example) users chose this book and title then return those details.

It doesn't sound feasible to me. Who would host this massive database? Does every user have to download it?

Look - I understand why you want more automation in data entry. So did I. That's why I contributed code to the project - to make the initial data entry job easier. It is currently much, much easier. The improvements from 0.6 to the new 0.7 are huge.

You mentioned entering a library of 14k books so even if only 5% of these books needed manually adjusting this would still mean 700 books need adjusting.

It was far more than that. I didn't write the merge record code until I was nearly done. I didn't write the option that lets you turn off "Overwrite author/title" until I'd lost a huge block of books and had to re-enter them (with that option on, a bulk metadata fetch may cause you to overwrite the correct author/title on a percentage of the books. Once lost, it becomes very hard to get that data back other than manually.) I didn't tackle the "If books with similar titles and authors found, merge the new files automatically" feature until I was more than halfway done. I did lots of manual fixing. Heck, I'm still fixing stuff. Some books don't have covers. Others have tags that need adjustment.

I suspect Calibre will continue to improve, but it will probably never reach the stage of completely automated metadata fetching.

I think it's great such options exist but I just can't see why a user needs to use them when the program could sort much of this out with a little fuzzy logic.

There are always improvements that can be made, but there are limited programming resources. If you want to contribute, Calibre has an easy to use development environment, and assistance is always welcome.

[mike_bike_kite]

You can easily delete books by highlighting them and pressing delete but is there a way to undelete books?


PS Good job on 7.1

[chaley]

Originally Posted by mike_bike_kite

You can easily delete books by highlighting them and pressing delete but is there a way to undelete books?

Sorry, but no, there isn't an undelete.