[The Older version] Calibre2opds V2

[dpierron]

I released a new version (0.0.6) which adds a lot of stuff :

• corrected (again) the database access
• added multiple file formats (all by default, use --includeformats)
• the user has the choice of options (use --help to find out which)
• added cross-references in the books pages (to other books by the same author, other books in the same categories, etc.)
• added links to Goodread in the books details (and to Wikipedia, but only in the HTML catalogs)
• rewrote the output engine, we are now OPDS-compliant (meaning that Aldiko users can also use our catalogs)
• added HTML catalogs alongside the XML ones, with nice popup menus (hint: catalog.html) (Ian's idea, thanks go to him)
• added generation of thumbnail images, to save bandwidth and speed up the feed loadings in Stanza
• stole the icons of Feedbook to use them in my feed (I hope it's not a problem for Hadrien...)
• all the catalogs are now paginated
• added the option of not generating the download section in the HTML catalogs (again, Ian's idea)

Please download it from the project page ; I hope you'll enjoy it, remember to report if you find any bugs.

Also, feel free to register an account at Sourceforge and submit bug reports, and support or feature requests, I'll be happy to implement them !

PS : I'd like to warmly thank Jane and Ian for their patience, their support, and their help !

[itimpi]

I tried the latest version against my library which has about 8000 books in it and got the following

11:06:53,521 INFO  [Catalog] Generating tags catalog
Exception in thread "Image Fetcher 0" java.lang.OutOfMemoryError: Java heap spac
e
        at java.awt.image.DataBufferInt.<init>(Unknown Source)
        at java.awt.image.Raster.createPackedRaster(Unknown Source)
        at java.awt.image.DirectColorModel.createCompatibleWritableRaster(Unknow
n Source)
        at sun.awt.image.ImageRepresentation.createBufferedImage(Unknown Source)

        at sun.awt.image.ImageRepresentation.setPixels(Unknown Source)
        at sun.awt.image.ImageDecoder.setPixels(Unknown Source)
        at sun.awt.image.JPEGImageDecoder.sendPixels(Unknown Source)
        at sun.awt.image.JPEGImageDecoder.readImage(Native Method)
        at sun.awt.image.JPEGImageDecoder.produceImage(Unknown Source)
        at sun.awt.image.InputStreamImageSource.doFetch(Unknown Source)
        at sun.awt.image.ImageFetcher.fetchloop(Unknown Source)
        at sun.awt.image.ImageFetcher.run(Unknown Source)
Exception in thread "Image Fetcher 0" java.lang.OutOfMemoryError: Java heap spac
e
        at java.awt.image.DataBufferInt.<init>(Unknown Source)
        at java.awt.image.Raster.createPackedRaster(Unknown Source)
        at java.awt.image.DirectColorModel.createCompatibleWritableRaster(Unknow
n Source)
        at sun.awt.image.ImageRepresentation.createBufferedImage(Unknown Source)

        at sun.awt.image.ImageRepresentation.setPixels(Unknown Source)
        at sun.awt.image.ImageDecoder.setPixels(Unknown Source)
        at sun.awt.image.JPEGImageDecoder.sendPixels(Unknown Source)
        at sun.awt.image.JPEGImageDecoder.readImage(Native Method)
        at sun.awt.image.JPEGImageDecoder.produceImage(Unknown Source)
        at sun.awt.image.InputStreamImageSource.doFetch(Unknown Source)
        at sun.awt.image.ImageFetcher.fetchloop(Unknown Source)
        at sun.awt.image.ImageFetcher.run(Unknown Source)

Any idea why this should be happening? Do you think there might be a problem managing large libraries with the current version of this tool? I was not sure what size libraries you had tested against? If you have no idea of the cause then I can try debugging it myself but thought I would ask first before going to that effort. Looking in Task Manager Java seemed to only be using about 85MB of RAM so there should be plenty more available.

Note that as well as being 8000 books, there will normally be about 4 formats for each books plus the cover so there are lot of files and folders in the library (total size on disk is ~12GB).

The system I was running on was using Window 7 Pro and has 2GB RAM.

[dpierron]

The problem is that Java, by default, only reserves 64Mb of RAM for its processes.

In your case (a very, very large library), this is not enough.

I made a correction, which will be included in the next release.

In the meantime, you should be able to run the tool if you edit the calibredbxtract.bat file, inside the bin subfolder of the directory where you installed the tool : replace the line showing set EXTRA_JVM_ARGUMENTS= with set EXTRA_JVM_ARGUMENTS=-Xmx256m

[itimpi]

Originally Posted by dpierron

The problem is that Java, by default, only reserves 64Mb of RAM for its processes.

Fair enough - I thought it might be something like that but did not off-hand know where to look to check.

In your case (a very, very large library), this is not enough.

There are others on the forum who have been talking about libaries 3-4 times the size of mine. It does mean that I do not use DropBox because of the amount of space it would require. Instead everything is hosted under Apache on one of NAS servers at home.

I made a correction, which will be included in the next release.

In the meantime, you should be able to run the tool if you edit the calibredbxtract.bat file, inside the bin subfolder of the directory where you installed the tool : replace the line showing set EXTRA_JVM_ARGUMENTS= with set EXTRA_JVM_ARGUMENTS=-Xmx256m

Thanks - I will try that.

Maybe you should add soething about the RAM requirements v Library size to the README file? You do not want to make the tool grab a lot of RAM for those who have much smaller libraries so simply telling people what to do if the error requires might be the way forward?

[dpierron]

Originally Posted by itimpi

Maybe you should add soething about the RAM requirements v Library size to the README file? You do not want to make the tool grab a lot of RAM for those who have much smaller libraries so simply telling people what to do if the error requires might be the way forward?

Well, technically you're correct, but I would end up generating multiple batch files to start the application with different RAM sizes ; in the end, the user would have to choose which one to run, and become aware of this very technical thingie, which he does not want to...

I think that 256 Mb of RAM is a very small chunk of most modern computers, especially with virtual memory ; also, the tool is meant to run a few times every hour or day, and will be running for 5 to 20 seconds on most computers.

So, IMHO, leaving this as the default value is perfectly acceptable, and the advanced users who know they don't need as much and want to minimize the tool's impact on their computer can edit the script...

Anyway, thanks for pointing out the problem, I have such a small library (176 books) that it would never have happened to me

Regards,
David.

PS: I took for granted that you used Windows, but the edit I mentionned works also in the Unix shell script that resides in the same directory, of course...

[itimpi]

With the RAM set to 256MB the tool completed without an issue (it looked a though peak RAm was about 110MB). However it took a LONG time, and generated 15314 files totalling 260MB in size!

Some comment on the output:

• The sorting seems a bit eratic:
  Authors seemed to be unsorted
  Series was fine (alphabetical ascending)
  All Books and then a ltter was in reverse alphabetical
  
• There probably needs to be another level of indirection in Series and Authors for large libraries (I have about 1200 series and 1100 authors). That was why I added that extra level in Calibre2Web. In fact I had though of adding even another level for the letters with a very large number of entries.
  
• In the HTML output, you say either "Download ePub file" , but for all other formats use the text "Download this ebook as XXX"
  
• You mention books as being "Book # in the xxx serie" (i.e. a missing s)

Having said that - the output is still useful - especially the HTML file option.

I have (finally) started getting seriously into the Python version that will be able to handle large libraries as the pages will be generated dynamically on demand directly from the metadata database. If you do not mind I will "borrow" some of the ideas from your implementation. One big advantage that dynamic pages give is the ability to support search functions as well as avoiding the time overhead of generating the static catalogs.

I am trying to write the Python is such a way that it can either be integrated into Calibre, or alternatively run as a free-standing piece of code under a web server such as Apache (so that calibre itself does not need to be running). Whether I can actually achieve that I am not yet sure. It may as a by-product also be able to output the static catalog files - do you have any objection to it possibly superseding your tool?

[macr0t0r]

Of course, one advantage of a thoroughly free-standing web-database such as this is that we Apple users can dump the output into our "Sites" folder on our iDisk and access it anywhere without running a server at all!

Calibre doesn't take much resource when it's just running the online library. What's the advantage of your Python code? Better customization? Or are you just trying to avoid having to run 2 servers? Since it's in Python, I wonder if there are some features that would be useful if rolled into Calibre.

- Jim

[dpierron]

You know what ? When I read and answered your first post about memory requirements, I didn't realize that it was you
If I had, I'd have started a conversation about the project !

Originally Posted by itimpi

With the RAM set to 256MB the tool completed without an issue (it looked a though peak RAm was about 110MB). However it took a LONG time, and generated 15314 files totalling 260MB in size!

Well, in a previous version I generated one file per book, and then linked to this file in the books lists (recent, authors, series, and all books).
I found out that Stanza did not support this feature of the OPDS standard, and then I rewrote everything to include the full book entry in the books lists.
This, of courses, makes for a much bigger output than the first solution...
And then, the thumbnail generated are, of course, a huge part in this big space...

Originally Posted by itimpi

The sorting seems a bit eratic:
Authors seemed to be unsorted
Series was fine (alphabetical ascending)
All Books and then a ltter was in reverse alphabetical

This is corrected in the next release.

Originally Posted by itimpi

There probably needs to be another level of indirection in Series and Authors for large libraries (I have about 1200 series and 1100 authors). That was why I added that extra level in Calibre2Web. In fact I had though of adding even another level for the letters with a very large number of entries.

You mean, an option to split these lists by letter, too ?

Originally Posted by itimpi

In the HTML output, you say either "Download ePub file" , but for all other formats use the text "Download this ebook as XXX"

Also corrected in the next release.

Originally Posted by itimpi

You mention books as being "Book # in the xxx serie" (i.e. a missing s)

I'm not sure I understand ? "Book 2 in the Giants serie" is meant as "the second book in the Giants serie" ; is this wrong ?

Originally Posted by itimpi

Having said that - the output is still useful - especially the HTML file option.

This one was actually Ian's idea, and I like it very much...

Originally Posted by itimpi

I have (finally) started getting seriously into the Python version that will be able to handle large libraries as the pages will be generated dynamically on demand directly from the metadata database. If you do not mind I will "borrow" some of the ideas from your implementation. One big advantage that dynamic pages give is the ability to support search functions as well as avoiding the time overhead of generating the static catalogs.

That's great (having news about the Python version at last), I can't hide that I began this project because I thought that your Python script development was cancelled !
And of course I don't mind you borrowing, I even would like to help if I can...
Maybe I can explain the pitfalls I managed to avoid, and if I can upgrade my Python-fu to second dan, I may even be able to help you code ?
To be honest, I coded this in Java to be quick, because I know this language very well. But I know a bit of Python, and I'd love to learn more, if you're patient enough

Originally Posted by itimpi

I am trying to write the Python is such a way that it can either be integrated into Calibre, or alternatively run as a free-standing piece of code under a web server such as Apache (so that calibre itself does not need to be running). Whether I can actually achieve that I am not yet sure. It may as a by-product also be able to output the static catalog files - do you have any objection to it possibly superseding your tool?

Again, of course, please reuse whatever you feel useful ; in fact, it may be cool to merge both projects, don't you agree ?

Feel free to contact me (dpierron at gmail dot com) about this, whatever your opinion about merging is ; in any case, I'll be more than happy to help and to share !

Cheers,
David.

[itimpi]

Originally Posted by macr0t0r

Of course, one advantage of a thoroughly free-standing web-database such as this is that we Apple users can dump the output into our "Sites" folder on our iDisk and access it anywhere without running a server at all!

That is OK - but it does not scale very well. It took me over 12 hours to generate the 15000 files (260MB in total size) to access my library.

Calibre doesn't take much resource when it's just running the online library. What's the advantage of your Python code? Better customization? Or are you just trying to avoid having to run 2 servers?

I have a NAS box which hosts my Calibre library and has a built in web server. The box has a built-in Apache server so I would like to run under that. The box is not capable of running Calibre directly.

Since it's in Python, I wonder if there are some features that would be useful if rolled into Calibre.

The intention if I get it right is that my enhanced server will replace the one currently in Calibre. That is why I am interested in it being in Python.

[itimpi]

Originally Posted by dpierron

I'm not sure I understand ? "Book 2 in the Giants serie" is meant as "the second book in the Giants serie" ; is this wrong ?

There is no singular of the word "series" - it must always have the final 's'.

That's great (having news about the Python version at last), I can't hide that I began this project because I thought that your Python script development was cancelled !

It was not cancelled - it just got put on hold as I had so much else on. The original idea of the VBScript version was that it was easy for prototyping what the format of the catalogs was, and also how they should be structured, particularily as the size of the library grew. The VBScript approach allowed me to use DAO which gave a nice abstraction to the database that I was very familiar with using.

The thought was that the Python version would follow quite quickly, but the Calibre2Web script worked well enough that there was not enough personal pressure to get on with the Python version. There was also the overhead of learning one's way around the exisitng Calibre code. Now that my library size is growing I find that the static catalog approach is becoming a bit unwieldy in terms of the time required to generate the catalogs.

And of course I don't mind you borrowing, I even would like to help if I can...
Maybe I can explain the pitfalls I managed to avoid, and if I can upgrade my Python-fu to second dan, I may even be able to help you code ?
To be honest, I coded this in Java to be quick, because I know this language very well. But I know a bit of Python, and I'd love to learn more, if you're patient enough

I am new to Python as well - which was one of the reasons it did not get done in that language originally. I am sure Kovid could write the code required in no time at all I have been playing with Python for the last few months and getting more familiar with it. The syntax is realtively easy as I am very familiar with C, C#, Jave and a variety of modern scripting languages. the issues seem to me to be (rather like Java) to find one's way around the vast number of libraries available.

Again, of course, please reuse whatever you feel useful ; in fact, it may be cool to merge both projects, don't you agree ?

Feel free to contact me (dpierron at gmail dot com) about this, whatever your opinion about merging is ; in any case, I'll be more than happy to help and to share !

You can also get me by email at itimpi at ntlworld dot com

Cheers,

Dave

[cistehovado]

Thank for useful tool, I have question, how can I edit or set text which is generated to html / xml pages, for example this text: "Alphabetical index of the single author - very useful indeed "

[dpierron]

Hi cistehovado, you're welcome !
There are two parts in my answer...
First, the tool is not internationalized yet (although it's on my todo list, and if I've got enough time before XMas, I'll make it in the next release).
When it will be internationalized, the texts will still not be user-editable, but if you want to translate the tool in your language, you'll be more than welcome !
At this point, all that will be needed will be to translate an english text file into whatever language you like, and send it back to me... Piece of cake, when you speak both english and that other language

Secondly, this cheesy sentence is there because there is only one author in your authors list ; this should not happen often, but maybe I could enhance the tool by skipping the authors list altogether when there is only one (and maybe, also extend this concept to other items ?)... I'll see what I can do.

Regards,
David.

[cistehovado]

I can translate it of course, but I need all text used in program.
I really do have only 1 author ;-) I thought that that message there will be always, but its kinda programmer joke I see ;-)

[dpierron]

Originally Posted by cistehovado

I can translate it of course, but I need all text used in program.

When the tool will be easily translatable, I'll post the translation files here, promised !

Originally Posted by cistehovado

I really do have only 1 author ;-) I thought that that message there will be always, but its kinda programmer joke I see ;-)

Yes, I thought it would almost never happen

[dpierron]

I released a new version (0.0.7) which adds and corrects a lot of stuff :

• * the books in the "books by letter" subcatalogs are now sorted by title
  * the links to Wikipedia in the "recent books" subcatalogs were including the date in the title
  * the eReader format should now be recognized by Stanza
  * the tool is launched with a 256Mb memory setting by default (useful if the user's library is huge)
  * added a warning when the thumbnail size is set to more than 125
  * added the thumbnail in the Stanza style (x-stanza-cover-image-thumbnail) because Stanza does not support this feature of OPDS
  * corrected the links (urlencoded them)
  * enhanced the HTML catalogs
  * added internationalization support
  * the tags catalogs can be categorized (i.e. if your tags are formatted like Action:ToRead:Now, they will be generated as a tree of catalogs)

Please download it from the project page ; I hope you'll enjoy it, remember to report if you find any bugs.

Also, feel free to register an account at Sourceforge and submit bug reports, and support or feature requests, I'll be happy to implement them !

Beginning in this release, it is possible to translate the tool in other languages than english. I already contributed a french translation, and I count on you guys to help me get really international

The translation files are attached to this post ; they should be self-explanatory, if not, feel free to contact me. When a translation is finished, send the files to me and I'll include them in the next release.

In these files, lines starting with a dash (#) are comments, and you have to escape (by doubling them) single quotes (i.e. ' becomes '')