Too good to be true

[sidd.artha]

I have been using Calibre for over 6 or 7 patches. And it has everything I wanted for my book colection and even a few things more. But everything stops when you end up using a test database of 3 books and move to the real thing. I have put a little over 2500 books, which is only part of the contents of my hard drive and everything went down with a bang.

Loading seems to go slower, although all it should do is read the internal database and let the rest for the time a users accesses information.

Once it loads it is huge in terms of memory. On a netbook with WinXP which has to fit in 1G of RAM you can't load something like a webbroswer and navigate as one would have to move to swap.

The autodetection is nice. But it takes only less than for ever. And there are some issues as it does not identify a book by its contents. It only tries to load the metadata stored infile. Say it does not find/read the metadata you get Unknown by Unknown and (!) you are prompted for duplicates. MD4 or MD5 are quick enough to help. Even a check of the filesize would show it's not a duplicate.

The last patch has shortened somehow the loading of books in the database, yet it will fail to read some files with an error.

And when it will fail with an error you have no way of knowing that the book WAS actually inserted in the database.

Which leads to another issue - for every tiny patch there is quite a file to be loaded. I'm sure at least the icon themes would not be that interesting to download over and over again. I believe that is a side effect of not having to pay for the actual bandwidth.

Also, there is an internal mechanism to access the net for upgrades, yet the upgrade has to be done with a browser. Why?

If there is the code for checking for upgrades, and the author has thrown in here everything but the kitchen sink (reader, converter, database) and there is a book reader which supports some HTML, yet for checking out the patched issues you get to fire the web browser. Which, as I said above, means a lot for a 1G machine. I'm sure that there are geeks out there with 1TB of RAM or even more, yet there are still out there laptops with 256MB or 512MB of RAM. They are quite good for having as book readers, as with my netbook, yet they could not possibly run an unoptimised mammoth like Calibre.

Although we're not talking about an internal database (say the bookmarks in Firefox), but a DB based on the filesystem it would be safe to assume that some changes are going to be made. The checking system is quite rudimentary. And slow. And it will not adapt to changes, you have to access some menu (can't say which as the metadata editor is stuck from some time, but I would like not to crash Calibre for fear of losing all my work). You access that command and than you sort the files for entries no lenght and erase those.

As pointed above Calibre can't find a duplicate even when it needs to know (for example if the Unknown by Unknown is the same book as the others with the same labels) so it could not do it for you either. Thus you need to use some duplicate file finder which, in turn, would make changes to the filesystem, which would disturb Calibre.

I have tried to make it slim by using or abusing the plugin system only to discover it is something... I don't know what, only not a plugin system. I see lists of plugins for handling different hardware ebook readers. I don't use any of them as I am missing the point - they are bulky, they have clunky interfaces, most of them give power to the firm that made them and not me, and, for the sake of turning an alleged 8000 pages I have to pay at least what I have paid for my netbook only to have less memory, more to fiddle and almost no other functionalty besides justify and an mp3 player. But that is another issue. My problem is that I can't unload those plugins. And others. What kind of a plugin is that that won't unload?

Getting metadata from the servers seems to work well, only that you have to close the book you are reading. It was very uncomfortable to do that as books that don't show up in major lists have to be entered by hand. Probably the author has assumed either that there are no books in the world that are not on amazon.com or that you should print each time you want to edit something. You can't do two things with the interface. So first you have to open the book. Than you have to enter the data in the meta editor. Than you have to kill the book reader. And only after that you can save the metadata. Fail to follow these precise steps in this particular order and you have to cancel, go back and retake the steps. And the book should not be opened as there is a need to rename the file which can't be postponed.

Saving the metadata for a 10 book database is quite ok. But saving the metadata for a 2500 book database means you have to wait about 30 minutes. At least that was my limit, after that I crash it. So far I found out that the data was written, but I always stay with the scare that everything would be lost. But as there is no cancel button (not even refresh so the metadata window will have ghosts of the other windows) I really can't see any other way.

There would be more things to report, but I already got tired of writing them down and you probably lost your patience to read this.

My conclusion: too good to be true. Wasting memory and a minute of loading time to manage a database of 10 books seems pointles. And when there's a large electronic bookcase the app becomes a pain. It rests the comfort of the convertor which I haven't tested so far, but which sports an impressive list of to/from formats. And the continuous list of supported news services which I hope they bring a good source of revenue to the author who could be able to make something of Calibre by the time it reaches version 1.0

[Lady Fitzgerald]

Oh, goodie. We have another one.

[EowynCarter]

Slowness isn't a problem I had with calibre. Guess I didn't went as far as that. Calibre was a life savior when i needed batch converting my mobi to ePub, but that's about it.

Then 2500 is quite a pack of books, guess the first import there will be long, no mattter the app.

[Starson17]

Originally Posted by sidd.artha

I have been using Calibre for over 6 or 7 patches. And it has everything I wanted for my book colection and even a few things more. But everything stops when you end up using a test database of 3 books and move to the real thing. I have put a little over 2500 books, which is only part of the contents of my hard drive and everything went down with a bang.

2500 books is a small database. I've seen databases more than 10 times that size. Mine is 7 times that size. Initial import into the database can take time, but it only happens once, and if you can find a program that does the initial import faster, go for it.

The autodetection is nice. But it takes only less than for ever. And there are some issues as it does not identify a book by its contents. It only tries to load the metadata stored infile. Say it does not find/read the metadata you get Unknown by Unknown and (!) you are prompted for duplicates.

It's not Calibre's fault if 1) your internal metadata sucks and 2) you didn't bother to read how to extract metadata externally from filenames using the regex import functions.

MD4 or MD5 are quick enough to help. Even a check of the filesize would show it's not a duplicate.

They wouldn't help. You're not trying to identify bit for bit dupes, you're trying to identify whether it's the same title book written by the same author. A PDF version is a duplicate of a LIT version if it's the same book. It's your obligation to get the right metadata into Calibre. If you don't want to bother, just let Calibre add all the books. The "Duplicates" message is just informational for you.

Saving the metadata for a 10 book database is quite ok. But saving the metadata for a 2500 book database means you have to wait about 30 minutes.

It sounds like you have an antivirus interfering. My 6-7 times larger DB saves in seconds.

I'm not going to try to convince you to use it. It's not for you. Go elsewhere. I have none of the problems you face, but then I don't run on a tiny machine.

[Leep]

I run my Calibre database of 1800 books on a netbook. Sure, it's not quite as fast, but still definitely adequate based on the things that it does.

Collects metadata from the web
Allows multiple formats for a given book
Converts to a variety of formats
Provides multiple ways to view your ebook collection
Allows plugins to deal with DRM
Connects to multiple devices
Has good drag and drop functionally

For me, this is a fabulous program that greatly simplifies organizing my ebook collection.

cheers

[kacir]

Dear sidd.artha

I think that there is something wrong with your system.
I have personal experience with a database several times larger than yours. Calibre DOES slow down with the number of books but not as drastically as your system.
Conversion from lit files takes way too long to my taste, so I use external tool, much quicker but not so sophisticated, but other than that I can't really complain.

Looking for duplicates works differently than you think. When I have Robinson Crusoe.epub already in database and when I drop in Robinson_Crusoe_-_D._Defoe.pdf I expect that the system will recognise newly added book as a "duplicate" and merge it with the existing record.

I prefer not to use autodetection, I usually use complicated Regular Expressions to get most of the metadata I am interested in from the filename.

I DO wish the Calibre was faster, especially with large number of books, but the slow-down is not as catastrophic as you experience.

[Starson17]

Originally Posted by kacir

Looking for duplicates works differently than you think. When I have Robinson Crusoe.epub already in database and when I drop in Robinson_Crusoe_-_D._Defoe.pdf I expect that the system will recognise newly added book as a "duplicate" and merge it with the existing record.

Merging into an existing record will only happen if the autosort/automerge option is turned on (In Preferences, Import/Export, Adding Books). That option looks for exact match on Author and similar Titles. By default Calibre will not "merge it with the existing record" and will simply identify the title as a duplicate and ask if you want to add it as a separate, new record, or not.

I prefer not to use autodetection, I usually use complicated Regular Expressions to get most of the metadata I am interested in from the filename.

I also use only the filename for metadata. Internal metadata is notoriously bad. (I'm not sure what you call "autodetection" - if it's detecting internal metadata, I agree it's better not to use that for many books. If it's the autosort/automerging option that puts the book into existing records, then you are using it.)

I DO wish the Calibre was faster, especially with large number of books, but the slow-down is not as catastrophic as you experience.

You should have used it 9 months ago. There have been numerous speed improvements. It's much, much faster now. There are inherent limits on the speed due to the need to support multiple OSs, but what can be done on speed will be done, it just takes time.

[kacir]

Originally Posted by Starson17

Merging into an existing record will only happen if the autosort/automerge option is turned on

Yes, this is exactly what I had in mind.
Using one of my Regular Expression "Add books" filters would identify those two books as identical

Originally Posted by Starson17

(I'm not sure what you call "autodetection" - if it's detecting internal metadata, I agree it's better not to use that for many books.

Yes, that is what I had in mind.

Originally Posted by Starson17

You should have used it 9 months ago.

I have been using Calibre from the very beginning - it was called LibPRS500 back then. And it was a lifesaver! I could use my brand-new new-fangled e-ink thingy - Sony Reader PRS-500 with my Linux PC.
I just haven't used it for serious library management. I started to play with library management functionality only a few months ago, when Calibre got UI overhaul (it is still not my cup of coffee, but Who Am I to complain? ;-) ) and when custom columns and serious Regular Expression support were implemented.

My book import RE is at the moment:
(?P<author>[^-]+)( - [[(]?(?P<series>[^-]+)[[( ]+(?P<series_index>[0-9]+)?[])]?)? - (?P<title>.+)

[Starson17]

Originally Posted by kacir

I have been using Calibre from the very beginning ...
I just haven't used it for serious library management.

That's why you may not have noticed the huge speed increases in the database functions. They relate only to library management.

[user_none]

Originally Posted by sidd.artha

Once it loads it is huge in terms of memory. On a netbook with WinXP which has to fit in 1G of RAM you can't load something like a webbroswer and navigate as one would have to move to swap.

It's a huge application with a lot of functionality. This is the same as complaining Photoshop uses too much memory.

Originally Posted by sidd.artha

The autodetection is nice. But it takes only less than for ever. And there are some issues as it does not identify a book by its contents. It only tries to load the metadata stored infile.

That's the point of metadata...

Originally Posted by sidd.artha

Say it does not find/read the metadata you get Unknown by Unknown and (!) you are prompted for duplicates.

You can have it use information from the filename if your metadata is bunk.

Originally Posted by sidd.artha

MD4 or MD5 are quick enough to help. Even a check of the filesize would show it's not a duplicate.

Books can have multiple files. This would cause every different format for a book show as a different book.

Originally Posted by sidd.artha

Also, there is an internal mechanism to access the net for upgrades, yet the upgrade has to be done with a browser. Why?

Calibre supports multiple OS's. Each one having different requirements for installing and updating applications. While an upgrade mechanism would be nice it's not a priority. It's a large difficult feature that will give little gain.

Originally Posted by sidd.artha

If there is the code for checking for upgrades, and the author has thrown in here everything but the kitchen sink (reader, converter, database) and there is a book reader which supports some HTML, yet for checking out the patched issues you get to fire the web browser. Which, as I said above, means a lot for a 1G machine. I'm sure that there are geeks out there with 1TB of RAM or even more, yet there are still out there laptops with 256MB or 512MB of RAM.

Again, would you run Photoshop on a machine like this?

Originally Posted by sidd.artha

They are quite good for having as book readers, as with my netbook, yet they could not possibly run an unoptimised mammoth like Calibre.

It runs fine on my 5 year old laptop. It also runs fine on Kovid's older computers.

Originally Posted by sidd.artha

As pointed above Calibre can't find a duplicate even when it needs to know (for example if the Unknown by Unknown is the same book as the others with the same labels) so it could not do it for you either.

Unknown by Unknown is the same book. A book is not a file, it can be multiple files (different formats).

Originally Posted by sidd.artha

I have tried to make it slim by using or abusing the plugin system... I don't use any of them as I am missing the point - they are bulky, they have clunky interfaces, most of them give power to the firm that made them and not me, ...

What? Supporting various hardware devices and showing their limitations empowers the form that made them and takes power away from you. If the device doesn't support what you want to do that's calibre's fault?

Originally Posted by sidd.artha

... and, for the sake of turning an alleged 8000 pages I have to pay at least what I have paid for my netbook only to have less memory, more to fiddle and almost no other functionalty besides justify and an mp3 player. But that is another issue.

That sounds like your main problem.

Originally Posted by sidd.artha

My problem is that I can't unload those plugins. And others. What kind of a plugin is that that won't unload?

A plugin system that dynamically loads the plugins when they are needed. If you don't plug in a Kindle the Kindle plugin will never be called and will never use any memory. All plugins are loaded on demand.

Originally Posted by sidd.artha

Probably the author has assumed either that there are no books in the world that are not on amazon.com

There are a number of metadata sources. Look at the plugins. Some require login information before they can be used.

Originally Posted by sidd.artha

And the continuous list of supported news services which I hope they bring a good source of revenue to the author who could be able to make something of Calibre by the time it reaches version 1.0

The news sources do not bring any revenue to the project. The news companies for the news sources do not pay to be in calibre.

[Worldwalker]

Originally Posted by sidd.artha

I have put a little over 2500 books, which is only part of the contents of my hard drive and everything went down with a bang.

I have 2500+ books and I don't seem to have this problem.

Admittedly I'm not trying to run calibre on a netbook. I'm not trying to run Photoshop CS5, either, which someone else used for a comparison. Calibre doesn't make a very good ebook reader.

Which leads to another issue - for every tiny patch there is quite a file to be loaded. I'm sure at least the icon themes would not be that interesting to download over and over again. I believe that is a side effect of not having to pay for the actual bandwidth.

You're not required to patch calibre. If you don't want to get a patch ... don't! As for what you believe, I can believe that the moon is made of green cheese, but that doesn't make it so.

They are quite good for having as book readers, as with my netbook, yet they could not possibly run an unoptimised mammoth like Calibre.

I believe you owe Kovid and the other devs an apology for your unwarranted assumptions and random insults.

Again, calibre is not meant to be a book reader to be put on a netbook. That's not what it's good at. It doesn't run on Linux-based readers, either. It's not for your ebook reader, any more than Photoshop is for your digital pictureframe.

Although we're not talking about an internal database (say the bookmarks in Firefox), but a DB based on the filesystem it would be safe to assume that some changes are going to be made.

I knew I was going to get to say this sooner or later: read my .sig.

I don't use any of them as I am missing the point - they are bulky, they have clunky interfaces, most of them give power to the firm that made them and not me, and, for the sake of turning an alleged 8000 pages I have to pay at least what I have paid for my netbook only to have less memory, more to fiddle and almost no other functionalty besides justify and an mp3 player. But that is another issue.

I have a netbook and an ebook reader. Of the two, the netbook is much bulkier. I love them both, for their own reasons, but I'd never mistake the netbook for the small, light, slim ebook reader. The ebook reader isn't bulky, it has a very streamlined, easy-to-use interface, the firm that made it has never interacted with it, it cost about half the price of the netbook (and I was a moderately early adopter), I don't care how much memory it has because my ebooks fit just fine, it has absolutely nothing to fiddle, and I don't want it to have any other functions, I want it to read ebooks, so I'm happy with it reading ebooks. If your problem is that your netbook isn't a full-scale laptop and isn't an ebook reader, I would propose the problem is not calibre; it's your choice of computers. Netbooks are wonderful and I wouldn't give mine up for the world, but I'm not going to mistake it for my laptop, or my desktop, any time soon. Or my ebook reader, for that matter.

My problem is that I can't unload those plugins. And others. What kind of a plugin is that that won't unload?

One that never loaded in the first place.

Getting metadata from the servers seems to work well, only that you have to close the book you are reading. It was very uncomfortable to do that as books that don't show up in major lists have to be entered by hand.

Or collected from the file name, which generally works moderately well.

Probably the author has assumed either that there are no books in the world that are not on amazon.com or that you should print each time you want to edit something.

His name is Kovid. It's on the calibre main screen, so you know it. He's reading this thread. You have just insulted him to his face. He was nice enough to write the free software you're using, and you're thanking him by insulting him. Nice.

And by the way, your insults, in addition to being rude, are also wrong. I get most of my metadata that isn't already in the book from places other than Amazon, and I've never printed anything from calibre in my life.

You can't do two things with the interface. So first you have to open the book. Than you have to enter the data in the meta editor. Than you have to kill the book reader. And only after that you can save the metadata.

I'm not sure what you're doing, or trying to do, but that's not the way other people edit their metadata. Most of us just ... y'know ... edit it. A lot of it in bulk. No opening and closing books, no printing, etc., involved.

My conclusion: too good to be true.

Imagining that you can run a full-featured desktop application on a netbook and have it behave exactly as it does on a desktop ... yeah, that's too good to be true. It's too good to be true of calibre, it's too good to be true of Photoshop, it's too good to be true of Dreamweaver, it's too good to be true of Age of Conan ... shall I go on? This may come as a bitter disappointment to you, but not every program in the world is written to run on your netbook.

For what it's meant for, calibre is wonderful.

For an ebook reader for your netbook, get something else. Calibre isn't it.

And the continuous list of supported news services which I hope they bring a good source of revenue to the author who could be able to make something of Calibre by the time it reaches version 1.0

You should be thankful that Kovid is a much nicer person than I am, because if I had mod powers here, you'd be out.

The news service recipes were created primarily by calibre users, the people who need to use them. Nobody gets any revenue whatsoever for them. In fact, the only thing Kovid gets from calibre is the occasional donation from its users. Did you donate? Or are you whining that a free program isn't exactly what you need?

And "make something of calibre"? You're apparently missing the million or so people who are quite happy with calibre exactly the way it is. They're probably not trying to run it on netbooks already hampered by XP. But, of course, you're ignoring all those people in your quest for bigger and better insults. You're really getting on my nerves now. If you want changes in calibre, you could always open a ticket and request them. Slinging insults at the developers will get you neither changes nor sympathy.

You owe Kovid Goyal an apology.

[Lady Fitzgerald]

I have calibre on my Asus netbook running XP and it works just fine. It's a bit slower than my desktop but that's just the nature of the netbook's Atom processor. Build a bridge and get over it.

[DoctorOhh]

Warning the below is a marginally useful response to a totally useless post. Probably best skipped by most.

For anyone coming across this post you canlearn about calibre here and skip the rest.

Originally Posted by sidd.artha

I have been using Calibre for over 6 or 7 patches. And it has everything I wanted for my book colection and even a few things more. But everything stops when you end up using a test database of 3 books and move to the real thing. I have put a little over 2500 books, which is only part of the contents of my hard drive and everything went down with a bang.

I have 4350 books and it works fine for me, but if your machine can't handle the load then move on or acquire a newer machine. I usually live at the trailing edge of technology too, but 2G of ram is the minimum I found useful for any set of apps.

Originally Posted by sidd.artha

Once it loads it is huge in terms of memory. On a netbook with WinXP which has to fit in 1G of RAM you can't load something like a webbroswer and navigate as one would have to move to swap.

Why does it "has to fit in 1G of RAM"? My daughter's netbook has 2G of ram and runs calibre fine. Most of the netbooks I checked out would let you go to at least 1.5Gs. Again if it doesn't work on your machine why bother us with your whining?

Originally Posted by sidd.artha

The autodetection is nice. But it takes only less than for ever. And there are some issues as it does not identify a book by its contents. It only tries to load the metadata stored infile.

Garbage metadata equals garbage results. This isn't too hard to understand. If your books have crap metadata then don't try to add them to calibre by having calibre search for metadata. Turn off get metadata by book and have it gather the Author and Title from the file name. You must have used some naming format to track your books. Check out the Preferences under Adding books.

Originally Posted by sidd.artha

Say it does not find/read the metadata you get Unknown by Unknown and (!) you are prompted for duplicates. MD4 or MD5 are quick enough to help. Even a check of the filesize would show it's not a duplicate.

This is checking for duplicate books not duplicate files. You should clean up your book directories and clear out duplicate files before you even think of adding your books to calibre.

Originally Posted by sidd.artha

The last patch has shortened somehow the loading of books in the database, yet it will fail to read some files with an error.

Garbage in garbage out, the PDF metadata reading/writing plugins are sometimes temperamental. These errors can be minimized by using metadata from filename on the initial loading of books. Also adding books in small batches will also minimize errors during adding books.

Originally Posted by sidd.artha

And when it will fail with an error you have no way of knowing that the book WAS actually inserted in the database.

The most recently added books pop into the top of the book list. If you doubt that then just click on the date column and sort the library by date added. The most recently added books will become apparent.

Originally Posted by sidd.artha

Which leads to another issue - for every tiny patch there is quite a file to be loaded. I'm sure at least the icon themes would not be that interesting to download over and over again. I believe that is a side effect of not having to pay for the actual bandwidth.

It's a side effect of Kovid having better things to do than create/support incremental upgrades for three Operating Systems.

Originally Posted by sidd.artha

Also, there is an internal mechanism to access the net for upgrades, yet the upgrade has to be done with a browser. Why?

You aren't even trying to figure this stuff out for yourself are you? There is no internal mechanism to upgrade. There is a mechanism that informs you a upgrade is available.

Originally Posted by sidd.artha

If there is the code for checking for upgrades, and the author has thrown in here everything but the kitchen sink (reader, converter, database) and there is a book reader which supports some HTML, yet for checking out the patched issues you get to fire the web browser.

So your whining that Kovid has created an entire suite of very useful features but doesn't support incremental upgrades for 3 different operating systems.

Originally Posted by sidd.artha

Which, as I said above, means a lot for a 1G machine. I'm sure that there are geeks out there with 1TB of RAM or even more, yet there are still out there laptops with 256MB or 512MB of RAM. They are quite good for having as book readers, as with my netbook, yet they could not possibly run an unoptimised mammoth like Calibre.

Again calibre runs fine on my daughters ASUS. You can always run the command line tools without opening the GUI for most functions.

Yada Yada Yada ... Yada Yada Yada

More useless whining. It is obvious you have decided to not run calibre, but instead of dealing with your decision like an adult you chose to come here and whine first. Your Mother would be so proud.

Originally Posted by sidd.artha

you probably lost your patience to read this.

True

Originally Posted by sidd.artha

My conclusion

Since you told us in the beginning of this post that you don't have a machine capable of running calibre any conclusion you voice would be of no value to anyone reading this forum. Now if calibre was a app you had to purchase before using then this tirade you have posted might be of use to those other folks living on the trailing edge of technology. As it is this tirade is a poor substitute for therapy.

Originally Posted by sidd.artha

And the continuous list of supported news services

The news recipes are primarily created by the user community.

You must have been tired by the end of your post because the whole "Conclusion" paragraph was a little disconnected and barely made sense.

[EowynCarter]

I'm a bit surprised, as the way calibre works (aka messing up your files), should makes it faster.

Because some of you are fine with the way calibre works on thier machine, don't mean reactivity shoudn't be improved...
This "calibre is perfect for my use, so it's perfect for everyone" is just anoying. Why such reactions everytime someone dare saying anything wrong about calibre ? Even when said harshly.

I was ok with sigil in version 1.2. Worked fine. Then 1.3 cames on, way faster. I didn't feel the "need" for improvement on the performance side, I still appresiated them. The first version of sigil woudn't have work on my 1G ram netbook, now it don't have any problem.

[user_none]

Originally Posted by EowynCarter

Because some of you are fine with the way calibre works on thier machine, don't mean reactivity shoudn't be improved...

Correct. However, the parent didn't come here asking for help nor did the parent provide constructive feedback.

For instance, the parent also eluded to the fact that Amazon wasn't able to find his books. Yet the parent also said that he looked though the plugins. If he had actually spend any time doing so he would have noticed calibre has metadata download plugins for six different places.

Originally Posted by EowynCarter

Why such reactions everytime someone dare saying anything wrong about calibre ? Even when said harshly.

The only thing I got out of the parents rant was, "It doesn't work on my super low end computer and I don't want to take the time to learn to use a complex piece of software. Make it work how I envision the world should work."

If there were concrete actual issues. Such as incorrect output during conversion there would have been plenty of people helping. If it was a bug in the output code who ever maintains that portion would have looked into why it was happening and would look into fixing it. However, that's not what the parent posted.