Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > Miscellaneous > Feedback

Notices

Reply
 
Thread Tools Search this Thread
Old 09-08-2008, 03:41 PM   #76
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by NetSlut View Post

Kovid, I recall you mentioned moving from a database to a file; can I ask what your reasoning was behind that, and how you feel it's worked out for you?

Originally, calibre stored both book metadata and the ebook files themselves in an sqlite database. This was so I wouldn't have to worry about the various file system quirks of 3 OSes. It has now moved to storing only the metadata in the database and the book files on the file system itself. This was done primarily to allow easy sharing of the calibre database between different computers over a network (the sqlite database can now be kept in memory for faster reads).

IMO, asking your users to run a separate database process is too much. Look into using some file based database engine like sqlite or metakit. Also if you use a database abstraction layer then users can choose any database backend to suit their needs.
kovidgoyal is offline   Reply With Quote
Old 09-08-2008, 03:44 PM   #77
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by NetSlut View Post
I was thinking that maybe I should write the metadata readers as separate entities, and put those in the open source community directly; so then we might get to a point where they can be maintained and added to by many people for many different programs, as long as the interface is consistent.
Note that calibre already has metadata readers/writers available as command line tools that should not bee to hard to interface with. You also need to give users a way of specifying how your program should interpret metadata from file names. This is because, contrary to the impression from this thread, most people actually dont use folder structures at all and just put metadata into structured file names.
kovidgoyal is offline   Reply With Quote
Advert
Old 09-08-2008, 03:46 PM   #78
Slite
Icanhasdonuts?
Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.
 
Slite's Avatar
 
Posts: 2,837
Karma: 532407
Join Date: Aug 2008
Location: Mölnbo, Sweden
Device: Kobo Aura 2nd edition, Kobo Clara HD
Quote:
Originally Posted by NetSlut View Post
One of the things that I think this approach would be useful for is in specifying the directory hierarchy of one's file system.

If you specify that your books are stored in directory structure platform/author/series/isbn/title, then I could create tag categories for each of those, and automatically populate the specific tags from the specific directories.

You're of course absolutely right that I'll need to read metadata from the ebooks themselves in order to more accurately populate certain categories (specifically title, I think, based on what I've seen so far), and of course for those ebooks that aren't in directory hierarchies at all.

I was thinking that maybe I should write the metadata readers as separate entities, and put those in the open source community directly; so then we might get to a point where they can be maintained and added to by many people for many different programs, as long as the interface is consistent.
The directory structure needs to be dynamic tho...

I tend to organize my books in /library/A/Last names begining with A, FirstName/FORMAT and so on and so forth....
Slite is offline   Reply With Quote
Old 09-08-2008, 04:39 PM   #79
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
Quote:
Originally Posted by kovidgoyal View Post
It has now moved to storing only the metadata in the database and the book files on the file system itself.
Yeah, that makes sense.

Quote:
IMO, asking your users to run a separate database process is too much. Look into using some file based database engine like sqlite or metakit.
That's what I'm feeling, although I don't like the performance of the file based ones. I might write something more tailored if the vote here is against a real database.

Quote:
Also if you use a database abstraction layer then users can choose any database backend to suit their needs.
Absolutely.
NetSlut is offline   Reply With Quote
Old 09-08-2008, 04:45 PM   #80
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
Quote:
Originally Posted by kovidgoyal View Post
Note that calibre already has metadata readers/writers available as command line tools that should not bee to hard to interface with.
Oh, that's fantastic. I'll definitely be using those, if you're ok with that?
Are they available as classes/libraries at all? That would be faster at run-time.

Quote:
You also need to give users a way of specifying how your program should interpret metadata from file names. This is because, contrary to the impression from this thread, most people actually dont use folder structures at all and just put metadata into structured file names.
Yes, that's a very good point too. That *will* be complicated. I don't like having to get users to type in regex expressions or funny syntax, so that'll take some real effort to come up with a suitable GUI.
NetSlut is offline   Reply With Quote
Advert
Old 09-08-2008, 04:47 PM   #81
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
Quote:
Originally Posted by Slite View Post
The directory structure needs to be dynamic tho...

I tend to organize my books in /library/A/Last names begining with A, FirstName/FORMAT and so on and so forth....
Definitely. I'm anticipating users initiating an indexing operation by specifying a range of locations to examine, and then specifying the directory/filename structure. It should persist only for that operation (but be retrievable, of course).
NetSlut is offline   Reply With Quote
Old 09-08-2008, 04:54 PM   #82
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by NetSlut View Post
Oh, that's fantastic. I'll definitely be using those, if you're ok with that?
Are they available as classes/libraries at all? That would be faster at run-time.


Yes, that's a very good point too. That *will* be complicated. I don't like having to get users to type in regex expressions or funny syntax, so that'll take some real effort to come up with a suitable GUI.
They're (mostly) python code, so unless you write your app in python or embed a python interpreter, you'll have to use them as commandline utilities.

Great if you come up with a good GUI, maybe I'll steal it from you (at the moment calibre requires users to enter regexps).
kovidgoyal is offline   Reply With Quote
Old 09-11-2008, 09:43 PM   #83
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
Thanks for all the feedback, fellows; I'm now designing away and figuring how to build this. I'll create a different thread when I have something to show you all, but I'll post a notice in here.
NetSlut is offline   Reply With Quote
Old 09-13-2008, 04:08 PM   #84
Slite
Icanhasdonuts?
Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.
 
Slite's Avatar
 
Posts: 2,837
Karma: 532407
Join Date: Aug 2008
Location: Mölnbo, Sweden
Device: Kobo Aura 2nd edition, Kobo Clara HD
Ohhh! Just figured out a function I'd really like to see. Have the program automaticly tell me which parts of a series I am missing if I ask it.

Pwett pwease? With a cherry on top...
Slite is offline   Reply With Quote
Old 09-13-2008, 04:19 PM   #85
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
That would be tricky. It's difficult to guarantee that I could automatically retrieve all parts of a series, so the only way I would be able to tell is if you entered the missing parts in the first place...
NetSlut is offline   Reply With Quote
Old 09-13-2008, 11:38 PM   #86
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by Slite View Post
Ohhh! Just figured out a function I'd really like to see. Have the program automaticly tell me which parts of a series I am missing if I ask it.

Pwett pwease? With a cherry on top...
FYI: That's a nice feature of LibraryThing and one of the main reasons I joined up. I wanted to know what Trek books I was missing from all the series they have it's hard to keep track.

Although, as NetS says, there really isn't any place to get this data. At LibraryThing it is all member supplied data. Although it looks like http://www.fantasticfiction.co.uk/ does have this type of info. Not sure if they have an API though.

BOb
pilotbob is offline   Reply With Quote
Old 09-14-2008, 08:27 AM   #87
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
Quote:
Originally Posted by pilotbob View Post
Although, as NetS says, there really isn't any place to get this data. At LibraryThing it is all member supplied data.
Yeah. I could maybe track some things down if I used a combination of APIs (from isbndb) and page scraping (searches from amazon, librarything, etc.) but it just wouldn't be very reliable.

I've just finally managed to hunt down the full list of The Destroyer series, for example, since no book shop has even the list, let alone the books.
NetSlut is offline   Reply With Quote
Old 09-15-2008, 01:22 AM   #88
Slite
Icanhasdonuts?
Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.
 
Slite's Avatar
 
Posts: 2,837
Karma: 532407
Join Date: Aug 2008
Location: Mölnbo, Sweden
Device: Kobo Aura 2nd edition, Kobo Clara HD
Quote:
Originally Posted by NetSlut View Post
Yeah. I could maybe track some things down if I used a combination of APIs (from isbndb) and page scraping (searches from amazon, librarything, etc.) but it just wouldn't be very reliable.

I've just finally managed to hunt down the full list of The Destroyer series, for example, since no book shop has even the list, let alone the books.
To bad, that would have been a truly awesome function.... Hmmm, maybe a way to add that manually? Or at least a way to add information about a series to the database?

Lets say that the program finds Eddings, David - Pawn of Profecy on my hd, that the software lets me enter Belgariad 01 as info for the book in question, that way one could at least sort on series and possibly have a function that lets me add a "wishlist"?
Slite is offline   Reply With Quote
Old 09-15-2008, 05:17 AM   #89
NetSlut
Connoisseur
NetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enoughNetSlut will become famous soon enough
 
NetSlut's Avatar
 
Posts: 99
Karma: 657
Join Date: Sep 2008
Location: Slough, UK
Device: Sony Reader
Quote:
Originally Posted by Slite View Post
Hmmm, maybe a way to add that manually? Or at least a way to add information about a series to the database?
Oh, that's absolutely in there.
Some people wanted to enter paper copies, for example, so what I figure is if you can add new books without requiring a file, then you can also tag them with "own" or not, and since "series" and "volume" tags are already present, that would combine nicely and automatically to do at least part of what you want.

The program is currently indexing books according to a series of meta-data reader plugins, using a user-specified directory hierarchy, and all based on tags. I'm starting the search facilities now.
The big thing it's missing amongst what it does do, is the GUI.

Won't be too long before I can get an alpha out, hopefully.
NetSlut is offline   Reply With Quote
Old 09-15-2008, 05:28 AM   #90
Slite
Icanhasdonuts?
Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.Slite ought to be getting tired of karma fortunes by now.
 
Slite's Avatar
 
Posts: 2,837
Karma: 532407
Join Date: Aug 2008
Location: Mölnbo, Sweden
Device: Kobo Aura 2nd edition, Kobo Clara HD
Quote:
Originally Posted by NetSlut View Post
Oh, that's absolutely in there.
Some people wanted to enter paper copies, for example, so what I figure is if you can add new books without requiring a file, then you can also tag them with "own" or not, and since "series" and "volume" tags are already present, that would combine nicely and automatically to do at least part of what you want.

The program is currently indexing books according to a series of meta-data reader plugins, using a user-specified directory hierarchy, and all based on tags. I'm starting the search facilities now.
The big thing it's missing amongst what it does do, is the GUI.

Won't be too long before I can get an alpha out, hopefully.
SAWEET! I'll definitly look forward to take this baby for a testrun!
Slite is offline   Reply With Quote
Reply

Tags
duck!, puppy!

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump


All times are GMT -4. The time now is 02:24 AM.


MobileRead.com is a privately owned, operated and funded community.