Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 10-31-2012, 12:46 AM   #1
orbatos
&nbsp
orbatos began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Aug 2012
Device: Me
Can The Base Structure Be Enhanced?

I have to agree on this issue in that I have the same problem and an addendum. The current inability to handle multiple Libraries, or even to include multiple libraries potentially from separate locations ( physically or logically ) is a large oversight that is hard to understand given the purpose and design of Calibre. Running a content server several times on the same machine on different ports is not a replacement for being able to access several libraries through one connection.

Related to this, the FAQ and periodic informational posts go on about how the folder structure is immutable and this is ok, even desirable, until you factor in the underlying file system. This is even acknowledged as a problem and yet there is no organizational method of circumventing it through use of containers or some other functionality despite the internal support for saw, zip, as a storage location.

For example, if the Author is the primary organizational point, why are the books stored in folders under this point rather than a container of some type, e.g. database or zip? reading manifests from these files would be far more efficient and less prone to problems than the current system, not to mention avoid Library data loss when working between different Operating systems or Calibre versions.

If the end user is not allowed to change these things, how will they be addressed?
orbatos is offline   Reply With Quote
Old 10-31-2012, 01:32 AM   #2
orbatos
&nbsp
orbatos began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Aug 2012
Device: Me
For comparison, The Project Gutenberg library is designed around machine accessibility and the accessibility of a large number of materials like a Calibre Library, but is arranged as follows:
Quote:
Originally Posted by http://www.gutenberg.org/wiki/Gutenberg:Mirroring_How-To
The main collection uses a set of numbered subdirectories. For example, eBook #12345 would have all associated files under 1/2/3/4/12345/. The generated content is all in one directory, with files named according to their eBook source: pg12345-images.epub and pg12345.txt.utf8. This is not "friendly" for users to browse, since it contains a very large number of files.
To be clear, I am not proposing the current structure be changed, but perhaps linkable Libraries ( or better awareness of alternates ) and the alternate use of containers rather than folders for data retention.
orbatos is offline   Reply With Quote
Old 10-31-2012, 01:53 AM   #3
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Update: I'll let others more knowledgeable respond to your questions.

Moderator Notice
Moved post to its own thread. Feel free to change the thread title to better reflect your question. If for some reason you can't do this let me know and I will change it to what you desire

Last edited by DoctorOhh; 10-31-2012 at 02:02 AM.
DoctorOhh is offline   Reply With Quote
Old 10-31-2012, 02:32 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,795
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Been a long time since one of these, "I dont like the way the calibre library works, change it for me threads". So to recap:

The calibre library structure is designed to be user browseable (and note, I said browseable, not modifiable). Originally, it was not, lots of people complained. That is not going to change.

And no, I am not going to maintain multiple library backends. If you want to know why, just look at the amount of code in the current library backend designed around trying to maintain data integrity for software that, unlike say Project Gutenberg, is designed to run in a huge number of contexts, by absolute laymen. We're talking millions of users vs. hundreds of users.

Dont use multiple libraries. The whole point of multiple libraries is, by definition, to separate the content in the libraries. There is zero point in creating multiple libraries and then demanding that calibre be able to combine those multiple libraries. That is not going to happen. Repeat after me, library A and library B means that the content of library A is separate from the content of library B. If you wish to partition your content, use the tools calibre provides for that purpose, namely, restrictions, in a single library.

And before I get someone complaining that they cant put all their book files into a single folder, note that you cannot list more than about 50,000 books at a time with acceptable performance. And 50,000 ebooks at approx 5MB per book is 250GB. Current hard drives are in the Terabyte range. So there's no good reason to not put them into a single folder.

Now, it is possible to come up with some examples where it is not optimal to have everything in a single folder. For example, I could be trying to maintain a library of 10,000 500MB PDF photo books. Or I could have badly organised disks, none of which individually have a lot of free space. But those are corner cases, calibre's design is not going to change to cater to them, at the expense of the vast majority of people for whom the design is optimal.
kovidgoyal is offline   Reply With Quote
Old 10-31-2012, 02:33 AM   #5
orbatos
&nbsp
orbatos began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Aug 2012
Device: Me
I'll apologize if my comments did not seem particularly clear, though they did not seem at all off topic to me, though maybe my understanding of the content server function is poorer than I realized.

As for my addendum in general I am surprised that I seem to be misunderstood, but I will make an attempt to break down some key issues.

Based not only on my own use, but also numerous complaints I have seen from others, Calibre seems to have problems with all of the following:
Loading/Organizing large numbers of materials ( admittedly a hard problem )
Inadvertent changes to the library ( even by automatic system processes )
File access ( especially where networks are concerned )
Storage ( primarily a file system issue )

As far as I have found, the only suggested solution to these issues has been to store everything locally, use multiple Libraries and to use the content server to access materials, neither of which address these issues at all. Feel free to correct me if any of this is untrue.

One suggestion made, perhaps unclear in purpose, was to allow parsing of containers other than folders as a method of reducing some of these issues. The reasoning is fairly simple, in that file system changes will not affect the internal structure of the container, and that performance might actually improve with the significantly reduced overhead of reading a manifest of a single file rather than individually accessing a minimum of 2 folders and 3 files per book.

Efficient alternatives to this somewhat more extreme than parsing a file as a storage location, if the Gutenberg Project comparison is any indication.

Edit: I will again say that I am not indicating a change in the structure of the Library, this point seems to be ignored too readily.
orbatos is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
The New Guy in Moon Base Twelve pigeonweather Self-Promotions by Authors and Publishers 0 04-12-2012 11:03 AM
Kindle installed base? BeccaPrice Amazon Kindle 3 11-07-2011 10:01 PM
LA question de base darakt Autres liseuses 0 10-30-2011 06:38 PM
[old thread] filename and library structure /author and titel structure tscamera Library Management 4 05-31-2011 05:44 PM
iLiad Possible to port DR base software to iLiad? ericshliao iRex Developer's Corner 0 03-12-2010 11:30 PM


All times are GMT -4. The time now is 05:45 AM.


MobileRead.com is a privately owned, operated and funded community.