Programming language for development

[ahi]

Originally Posted by kovidgoyal

A good point about RTF, I forgot to mention that. RTF conversion is slow in calibre, but that's largely because of a rather badly designed RTF processing library calibre uses. I've always been meaning to swap it out for another, but never get around it. Any volunteers

Hmmm... not as such, no. But since I will be, out of necessity, trying to do a moderately complete RTF 1.0 parser, let me ask what output does Calibre expect?

Perhaps if my needs and that of Calibre are not too far apart, I'll pass the relevant code your way for you to customize and fit into place at your leisure.

- Ahi

[tompe]

Originally Posted by HarryT

Conversion speed from Mobi to ePub seems to be remarkably inconsistent. All my books are created via the same route: BD -> HTML -> Mobi Creator -> Calibre, and yet sometimes I have two books which are pretty much the same size, and not obviously of different "complexities", and yet one will convert in 15 seconds and the other will take 15 minutes. Not a problem - just an observation .

Well, it is a problem if you want to batch convert all your book when a new and better version of Calibre is released.

[ahi]

Originally Posted by HarryT

sometimes I have two books which are pretty much the same size, and not obviously of different "complexities", and yet one will convert in 15 seconds and the other will take 15 minutes. Not a problem - just an observation .

The unfortunate collision of "coffee breaks" and "a unionized environment"?

- Ahi

[kovidgoyal]

Originally Posted by ahi

Hmmm... not as such, no. But since I will be, out of necessity, trying to do a moderately complete RTF 1.0 parser, let me ask what output does Calibre expect?

Perhaps if my needs and that of Calibre are not too far apart, I'll pass the relevant code your way for you to customize and fit into place at your leisure.

- Ahi

calibre expects HTML output. Basically it will give the parsing library the rtf file and expect it to return an HTML file with optionally metadata like table of contents etc. Preferred format for metadata return is OPF+NCX but I can work with something else.

[kovidgoyal]

Originally Posted by HarryT

As I said, I have no "complaints" at all - the speed is generally perfectly acceptable, and I've never known anyone fix reported bugs as rapidly as you do, so kudos for that!

Conversion speed from Mobi to ePub seems to be remarkably inconsistent. All my books are created via the same route: BD -> HTML -> Mobi Creator -> Calibre, and yet sometimes I have two books which are pretty much the same size, and not obviously of different "complexities", and yet one will convert in 15 seconds and the other will take 15 minutes. Not a problem - just an observation .

Offhand I'd guess that some of those books have "repeated but slightly different css". calibre's MOBI input plugin geenrates CSS from the MOBI attributes and when it does this, it tries to restrict the amount of duplicate CSS it generates (to speed up the eventual CSS parsing). However, some MOBI files have a lot of CSS that is different textually, but renders the same, leading to long parsing times.

[ahi]

Originally Posted by kovidgoyal

calibre expects HTML output. Basically it will give the parsing library the rtf file and expect it to return an HTML file with optionally metadata like table of contents etc. Preferred format for metadata return is OPF+NCX but I can work with something else.

Ok. No promises, but I'll let you know if I have something that might work.

[kovidgoyal]

Cool, thanks.

[ahi]

Originally Posted by kovidgoyal

Cool, thanks.

On a barely related note... is there a way to increase Python's memory limits? I keep getting generic and cryptic "MemoryLimit" errors when trying to process RTF files between 400 MB and 1 GB in size.

While this may not be shocking, I do have 3 GB RAM and 32 GB swap space... and only about 6 GB of memory (close to 3 GB RAM, and a bit over 3 GB swap) seems to be used before Python bails.

- Ahi

[kovidgoyal]

I think that will depend on your kernel. For example, historically linux kernels were configured so that user space processes (like python) could not address more than 2GB memory. IIRC the same is true of WinXP. More recent linux kernels are usually configured with higher mem limits and I'm note sure what the situation is in modern windows land.

Basically, python has no particular memory limits, it will happily consume all the memory the OS hosting ti will allow it to.

[ahi]

Originally Posted by kovidgoyal

I think that will depend on your kernel. For example, historically linux kernels were configured so that user space processes (like python) could not address more than 2GB memory. IIRC the same is true of WinXP. More recent linux kernels are usually configured with higher mem limits and I'm note sure what the situation is in modern windows land.

Basically, python has no particular memory limits, it will happily consume all the memory the OS hosting ti will allow it to.

Oh, so it's more likely (certain?) to be an OS imposed memory limit.

I guess I should just rewrite things a bit to process one chunk at a time.

Thanks!

- Ahi

[=X=]

Hi Pablo,
I think one should choose the language on what you're trying to accomplish. As you are familiar with different programming languages you're aware that each language has it's strength and weakness. Once you have a better definition of your requirements, then ask what languages is best for the task.

=X=

[HarryT]

Originally Posted by kovidgoyal

I think that will depend on your kernel. For example, historically linux kernels were configured so that user space processes (like python) could not address more than 2GB memory. IIRC the same is true of WinXP. More recent linux kernels are usually configured with higher mem limits and I'm note sure what the situation is in modern windows land.

32-bit Windows can address up to 3.5GB of physical RAM, and allocate up to 3GB of "user" memory (physical and/or virtual) to a process. 64-bit Windows has no effective limit on the amount of RAM it can use, but a 32-bit process still has the same memory limits.

[radius]

Originally Posted by kovidgoyal

Ah everyone's favorite debate. If you want to get into it seriously, I suggest being more precise. Whey you say calibre is "slow" waht do you mean?

1) Some people complain that the GUI is slow. Other people say taht the GUI is fast enough.
2) Some people claim it is slow with large databases, other people say it is fast enough.
3) Some people say it is slow with conversion, other people...
4) Some people say it is slow with news download, ...
5) Startup times are slow

If it bugged me enough I'd probably open an issue on your tracker (which I have done before) but a lot of things which I would expect to be pretty much instant seem to take multiple seconds.

Some examples (not including startup time which you've already noted):

- I only have 818 books in Calibre but when I click on the sort keys for the table of books (e.g. Title, Author(s), Size(MB) etc) it takes about 2 seconds for the sort to happen. In iTunes where I have far, far more songs, the sort seems to happen as soon as I let go of my mouse button.

- When I edit the metadata for a book, it takes about 3-4 seconds for the metadata dialog to go away.

- When I click on "Add books" or "Preferences" in the main button bar on top, the new dialog windows that pop up can be seen to draw. That is, the frames are visible and then the window filled in. I would expect that to be close to instant.

- Removing a book takes 1 second or more when I'd expect it to be instant

- After dragging a book from my desktop into the Calibre table of books to import it, Calibre takes about 3 seconds to finish reading metadata even for simple file types like text which don't have any.

- It would obviously be nice if conversion were faster, but this is the one place I would far prefer fidelity and quality over speed (but please work on making it faster anyway )

In general, I think Calibre has had enough features for me many, many releases ago. I would rather see UI annoyances go away first, and the sluggish feel of Calibre to me is one such annoyance.

That said, I think Calibre is great, and the best library manager/e-book format converter collection that I have tried so far.

[kovidgoyal]

The Edit metadata window goes away slowly because after you edit metadata, calibre has to move the book files to a new location.

Sorting will become faster after the database refactoring.

Seeing the windows drawing is definitely not normal, even on my 6+ year old laptop that doesn't happen.

Adding books has a floor time because it takes place in a separate process. This is for robustness as sometimes metadata reading libraries written in C can cause segfaults which would take down all of calibre.

[Pablo]

Originally Posted by =X=

Hi Pablo,
I think one should choose the language on what you're trying to accomplish. As you are familiar with different programming languages you're aware that each language has it's strength and weakness. Once you have a better definition of your requirements, then ask what languages is best for the task.

=X=

Rigth now I haven't decided what problems to address, but as they will be in the same line as utilities and programs found here in MR, I wanted to know the opinion of the experts.