Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 03-18-2009, 03:02 PM   #1
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
Can I autofetch info about my books?

Hello fellow readers! I have decided to begin using Calibre again after I found a great RegEx to import the metadata from my book titles.

My collection is in the crap format of pdf. I plan on using calibre to bring to them over to Epub. My collection is diverse - and largely inherited. Some are from Project Gutenberg, others from the various free libraries available. When I was less wise, I turned them all into the pdf format. And promptly killed all metadata, although there was little of it anyway.

Anyway, my collection is has the series, title, and author as metadata in their names. I will be importing them into calibre with the above mentioned RegEx, but they number a bit over a thousand. I tried it before - and calibre eventually was spending about an hour a book to import. (OK, not quite that long) So I want to make the process go as fast as possible this time. I've disabled all unnecessary plugins, and the searching for metadata inside files because I don't need it to. But still it takes a lot of time. What else can I do to make the books import faster?

My final issue is that all importing of metadata is done manually. I don't want to have to do that by hand - my titles, authors and series are correct, so I trust the search engine to select the right one. Would you consider adding an automatic way of importing that data?

And just one more question - what fields of the epub format can calibre fill in?

Thanks.
darkmonk is offline   Reply With Quote
Old 03-18-2009, 03:10 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Just break the input into batches of about a 100 to make it go faster.

Automating metadata import can be done using the calibre command line tools isbndb and calibredb by writing a small script to combine them

calibre fills in all metadata fields of an epub book and several that epub does't support as well
kovidgoyal is offline   Reply With Quote
Advert
Old 03-18-2009, 04:08 PM   #3
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
LOL, you sure out did yourself!
I laugh as I disable all the plugins and extra features. When I started to import the books, it was going blindingly fast. Now, about 40% in, it is going moderately fast. Hey, I finally found a program that uses more RAM then Firefox! Still getting about 10/min. Strange, no books have been copied yet.

OK, I finished reading the metadata - All the books are now being imported into Calibre's essentially proprietary directory structure. (JK)
This is going to take a while...

So in the mean time, about those scripts... Damn windows - I want my man pages back! I have no idea how to do that. Ah, I found your documentation. Looks like the GUI is a far better way of doing things. Seriously - I nominate an I'm Feeling Lucky button as the best feature Calibre could add

So - I believe the line
librarything -u darkmonk -p passwd *
would fetch all the cover art?
Meanwhile, I have no idea how to do so with isbndb because it requires multiple variables. Care to give me a bone?

If not, I'll try to make a macro to do it with each - and just let it run for a while...
darkmonk is offline   Reply With Quote
Old 03-18-2009, 04:10 PM   #4
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by darkmonk View Post
OK, I finished reading the metadata - All the books are now being imported into Calibre's essentially proprietary directory structure. (JK)
Why the sad face? If you don't care how Word stores your words in it's Doc file... why should you care how claibre stores its files?

BOb
pilotbob is offline   Reply With Quote
Old 03-18-2009, 04:28 PM   #5
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
Quote:
Originally Posted by pilotbob View Post
Why the sad face? If you don't care how Word stores your words in it's Doc file... why should you care how claibre stores its files?

BOb
That's a damn fair question. Here's my answer: I do care -which is why I don't use word , and love epub (because of how it stores my files )
I tried out Gobo linux a while ago. It's whole idea is to ditch databases - because directories were supposed to be the database. Makes perfect sense to me.

But everyone in a while, I can put up with a little bit of apple-like secrecy, and "just let things work..."
darkmonk is offline   Reply With Quote
Advert
Old 03-18-2009, 04:56 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by darkmonk View Post
That's a damn fair question. Here's my answer: I do care -which is why I don't use word , and love epub (because of how it stores my files )
I tried out Gobo linux a while ago. It's whole idea is to ditch databases - because directories were supposed to be the database. Makes perfect sense to me.

But everyone in a while, I can put up with a little bit of apple-like secrecy, and "just let things work..."
Huh? Do you even understand what you are saying? "Apple like secrecy", "proprietary directory structures"? What secrecy? Any directory structure is "proprietary", because some one has to specify what that structure is. If you think that storing books in your own directory structure is less proprietary you dont understand the meaning of the word proprietary.

And please point out to me one single use case for which storing books in your own directory structure is better.
kovidgoyal is offline   Reply With Quote
Old 03-18-2009, 05:40 PM   #7
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
Oh, dear... I was afraid my statement would blow out of proportion. I said I was joking... I know and fully understand that calibre, being open source, is 100% unproprietary. Yes, naturally you are correct - all directory structures are, in a senses, proprietary. No harm, or offense, meant.

But since I can't resist... I will name the one case where my own structure is better - when I want to find a file through a file manager. Naturally, though, Calibre is superior.

But could you please verify whether or not my line
librarything -u darkmonk -p passwd *
will fetch all the artwork?
darkmonk is offline   Reply With Quote
Old 03-18-2009, 05:46 PM   #8
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by darkmonk View Post
But since I can't resist... I will name the one case where my own structure is better - when I want to find a file through a file manager.
Calibre stores the books in folders organized by Author then by book. How is that hard? (Just playing devil's advocate here.)

BOb
pilotbob is offline   Reply With Quote
Old 03-18-2009, 05:48 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No, it wont you have to call

librarythin -u x -p y ISBN

with each ISBN
kovidgoyal is offline   Reply With Quote
Old 03-18-2009, 05:58 PM   #10
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
Quote:
Originally Posted by pilotbob View Post
Calibre stores the books in folders organized by Author then by book. How is that hard? (Just playing devil's advocate here.)

BOb
Oh, how little value you stick on a click... Why, my preferred habit happens to be sticking all the books by an author in one folder! That saves me one click everytime I open a book. Given enough time, I could have built a tower with the time it took me to do those clicks...
</end stupidity>

OK, I understand how that is working now. I would have to pipe the ISBN into each command, which means I need to get the ISBN first... I believe the line
isbndb -a x -t y
would pull the data down. So I would have to pipe a lot of info around, as well as remove it from Calibre's database. OK, that officially passed my skill level. Macros, here I come...

Still, I don't think I'm the only one who would like a "I'm Feeling Lucky" button added to Calibre. Just keep it in mind - I know you have a waiting list a mile high.
darkmonk is offline   Reply With Quote
Old 03-18-2009, 05:59 PM   #11
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by darkmonk View Post
Oh, how little value you stick on a click... Why, my preferred habit happens to be sticking all the books by an author in one folder! That saves me one click everytime I open a book. Given enough time, I could have built a tower with the time it took me to do those clicks...
I guess I am spoiled by Spotlight.

BOb
pilotbob is offline   Reply With Quote
Old 03-18-2009, 06:39 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by darkmonk View Post
Oh, how little value you stick on a click... Why, my preferred habit happens to be sticking all the books by an author in one folder!
Actually, calibre will open up the folder containing your books file if you press the O key (or right click on the entry in the book list, you you dont even have to click on the author folder.

Quote:
Still, I don't think I'm the only one who would like a "I'm Feeling Lucky" button added to Calibre. Just keep it in mind - I know you have a waiting list a mile high.
Sure, I have no objections to doing that. Just very little available time
kovidgoyal is offline   Reply With Quote
Old 03-18-2009, 06:39 PM   #13
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 5,111
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
I don't like Calibre's folder structure either. However, Calibre doesn't delete the original files and ebooks are relatively small so I just keep multiple copies. One is a set of books organized in my preferred folder architecture and the other set, the Calibre library. There's nothing stopping anyone from doing that. Even if you have 10 thousand books at 2MB/book (a very generous estimate), that's still just 20GB (40GB for 2 copies). Nowadays, you can get 1TB hard drives for less than $100.
ilovejedd is offline   Reply With Quote
Old 03-18-2009, 10:03 PM   #14
astrodad
Guru
astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.astrodad ought to be getting tired of karma fortunes by now.
 
astrodad's Avatar
 
Posts: 962
Karma: 568242
Join Date: Dec 2007
Device: Kindle Oasis 3, Kindle Paperwhite 1, iPad, iPhone
Quote:
Originally Posted by ilovejedd View Post
Nowadays, you can get 1TB hard drives for less than $100.
True, but only very bad ones that you wouldn't want as your primary storage device for so many files. Trust me. Been there, done that.
astrodad is offline   Reply With Quote
Old 03-18-2009, 11:13 PM   #15
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
First let me apologise for the somewhat pedantic tone of this post, if it seems like I feel I know all the answers, it's because I do

Let's take a step back and ask a slightly metaphysical question:

"Why do you store files in folders?"

I'm betting the answer to that is some version of the following two themes

a) I want to be able to find them easily
b) I want to categorize them

Now lets address these themes vis-a-vis calibre vs. the file system

a) When you want to find a file, 99% of the time, what you actually want is the book that file contains. In other words rather than searching for a file you are searching for a book title or author or genre or whatever. Now that kind of search is significantly more efficient when using a dedicated interface for it, like calibre.

b) Folders are really just a system of hierarchical tags. So when you put a file in the folder /mystery/asimov/foundation/ what you're saying is that the file is a mytery written by asimov and part of the foudation series. But the book could in general belong to many overlapping categories. For example the book is also scifi and future history and a hugo winner and written by a dead author and so on. There is no way you can express these kinds of relationships using hierarchical tags, but you can using arbitrary, i.e. non hierarchical tags, which is what calibre gives you.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
req: autofetch/list for books with missing metadata athlonkmf Calibre 7 10-27-2014 04:41 AM
Info Amazon collects about books I didn't buy from them? rangepig Amazon Kindle 1 08-26-2010 01:17 AM
Easist way to modify IMP books internal header info? askyn IMP 23 04-09-2009 10:20 AM
Info pbananth Which one should I buy? 8 01-30-2008 04:42 PM


All times are GMT -4. The time now is 03:08 AM.


MobileRead.com is a privately owned, operated and funded community.