[The Older version] Calibre2opds V2

[Xenophon]

Any news on finalization of the latest calibre2opds version?

Xenophon
(who is eagerly awaiting the next rev...)

[whodean]

<---has been awaiting a Mac version of the beta

[Nine989]

Hi


I´ve been using Calibre2opds and dropbox without any problems until I started getting error 403 every time I tried to browse the online catalogue. Does anybody know why this happens? All the files are on the public folder...

[electronicfur]

When running calibre2opds on my small NAS server it promptly crashed the server by using up all of the available memory.

When I run it on my pc I noticed during the loading & filtering database stage the java process grew to 500MB of memory. Does it read the whole library into memory?

I was hoping to set this up as a nightly cron job on my small NAS server. Is there any way to run it to consume less memory?

On another note, is there a way to use custom-meta data instead of tags? I know I'm not alone in that I use a Genre field (eg Science Fiction, Romance, Autobiography, etc) to classify my books. In this way I can leave the tags as set by the metadata download, but still classify my books by my own Genre definition.

Cheers,
EF.

[itimpi]

Yes - during the first stage all critical infomation from the calibre database is loaded into memory structures. Having said that, 500MB sounds extreme - on my own library with around 13.5K books all in several formats only about 120MB is used. Changing this approach is not practical as it would involve a complete rewrite from scratch. Having said that it is always worth revisiting the code to see if reductions can be made.

As to using custom fields this on the 'todo' list, but so far nothing satisfactory appears to be close to reaching completion. What is most likely to materialize is the ability to have a tag like named field treated in the same way as is currently supported for tags.

[electronicfur]

Thanks for answering my questions. I have about 10k books in my library, mostly in mobi or pdf format. I'm running on 64 bit windows, so maybe that accounts for some extra memory usage.

My NAS is only a plug computer with 128MB memory, so I guess I can forget running on that.

Does anyone know what the following error means? I get it with about 2000 of my books, and it results in empty book*.html and xml files. I have looked in the temp directory when calibre2opds is generating and the xml files causing problems are 0 bytes, which explains the reason for the error. But I cant figure out why they are zero bytes, as this is the first WARN level message in the log. And the books in question look fine in the Calibre library.

7-08-2011 23:41:29,881 [] WARN com.gmail.dpierron.calibre.opds.BooksSubCatalog - java.lang.NullPointerException
07-08-2011 23:41:29,881 [] WARN com.gmail.dpierron.calibre.opds.BooksSubCatalog - ... [3120 - L. Neil Smith]: Lando Calrissian and the Starcave of ThonBoka
07-08-2011 23:41:29,884 [] ERROR com.gmail.dpierron.calibre.opds.HtmlManager - cannot transform C:\Users\menno\AppData\Local\Temp\calibre2opds5234 373867683199796\_catalog2\book\book_16263.xml
javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: java.lang.IllegalStateException: Root element not set
at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transform(Unknown Source)
at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.transform(Unknown Source)
at com.gmail.dpierron.calibre.opds.HtmlManager.genera teHtmlFromXml(HtmlManager.java:55)
at com.gmail.dpierron.calibre.opds.HtmlManager.genera teHtmlFromXml(HtmlManager.java:33)
at com.gmail.dpierron.calibre.opds.BooksSubCatalog.ge tBookEntry(BooksSubCatalog.java:683)
at com.gmail.dpierron.calibre.opds.BooksSubCatalog.ge tListOfBooks(BooksSubCatalog.java:398)
at com.gmail.dpierron.calibre.opds.BooksSubCatalog.ge tListOfBooks(BooksSubCatalog.java:191)
at com.gmail.dpierron.calibre.opds.SeriesSubCatalog.g etSeries(SeriesSubCatalog.java:440)
at com.gmail.dpierron.calibre.opds.SeriesSubCatalog.g etContentOfListOfSeries(SeriesSubCatalog.java:213)
at com.gmail.dpierron.calibre.opds.SeriesSubCatalog.g etContentOfListOfSeries(SeriesSubCatalog.java:111)
at com.gmail.dpierron.calibre.opds.AuthorsSubCatalog. getAuthor(AuthorsSubCatalog.java:383)
at com.gmail.dpierron.calibre.opds.AuthorsSubCatalog. getListOfAuthors(AuthorsSubCatalog.java:197)
at com.gmail.dpierron.calibre.opds.AuthorsSubCatalog. getListOfAuthors(AuthorsSubCatalog.java:190)
at com.gmail.dpierron.calibre.opds.AuthorsSubCatalog. getSubCatalogEntry(AuthorsSubCatalog.java:457)
at com.gmail.dpierron.calibre.opds.Catalog.createMain Catalog(Catalog.java:646)
at com.gmail.dpierron.calibre.gui.Mainframe$2.run(Mai nframe.java:258)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.xml.transform.TransformerException: java.lang.IllegalStateException: Root element not set
at com.sun.org.apache.xalan.internal.xsltc.trax.Trans formerImpl.getDOM(Unknown Source)
... 17 more
Caused by: java.lang.IllegalStateException: Root element not set
at org.jdom.Document.getContent(Document.java:408)
at org.jdom.output.SAXOutputter.output(SAXOutputter.j ava:612)
at org.jdom.transform.JDOMSource$DocumentReader.parse (JDOMSource.java:518)
at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCD TMManager.getDTM(Unknown Source)
at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCD TMManager.getDTM(Unknown Source)
... 18 more

Cheers
EF

[electronicfur]

After trying to debug my problem, it did occur to me that there could be a few potential optimisations. My apologies if these are obvious and have already been considered, but I thought I would throw them out there in case they are useful.

I'm a bit confused for the need to read the whole database structure into memory. I thought the most efficient approach would be to process one book at a time, and I thought this was one of the advantages of calibre2opds over the inbuilt calibre opds server. I couldn't think of a reason why the whole structure is needed to be known up front.

I notice that a large chunk of time is spent on the "copy the library" part. The copy operation re-creates all the files already in the temporary directory again. This could be entirely avoided in most use cases by using a file move operation rather than file copy of the temporary files. The use case where calibre2opds is running on the same machine as the calibre library is the obvious one where a move operation would avoid recreating all the files again. And the use case of calibre2opds running against a network file share could avoid the file recreation if there was an option to specify the temporary file directory, so it could be placed on the same network share.

Finally I was unsure what the "Minimise number of changed files" option does.
I had thought that if calibre2opds read the date & time of the original _catalog creation and also placed the CRC of the profile.xml used to generate it in the _catalog, then upon start it could see if the profile.xml being used is identical, and use the date & time stamp to only process books that have been amended after this date & time. In my case its seems to always process all books, and only cary out a crc check on the generated files afterwards. Also for some reason my synclog.log always says "IGNORED (CRC)" for the files.

Cheers,
EF

[itimpi]

Originally Posted by electronicfur

I'm a bit confused for the need to read the whole database structure into memory. I thought the most efficient approach would be to process one book at a time, and I thought this was one of the advantages of calibre2opds over the inbuilt calibre opds server. I couldn't think of a reason why the whole structure is needed to be known up front.

The reason for doing this is it something like 10-100 times faster than going back to the database to get the details of each book. The decision was therefore made at implementation stage to take the memory hit to try and maximise the generate speed. Note that the big gain over the Calibre server is that you do not take the memory hit when trying to download books.

I notice that a large chunk of time is spent on the "copy the library" part. The copy operation re-creates all the files already in the temporary directory again. This could be entirely avoided in most use cases by using a file move operation rather than file copy of the temporary files. The use case where calibre2opds is running on the same machine as the calibre library is the obvious one where a move operation would avoid recreating all the files again. And the use case of calibre2opds running against a network file share could avoid the file recreation if there was an option to specify the temporary file directory, so it could be placed on the same network share.

It might be worth re-vistiing this code again to see if there is further room for optimisation. The current optimisation was added as extra logic that can be run after the main generate phase had been run. Note that the optimisation does NOT copy files from the temporary location to the final target if they are unchanged from the previous run.

The reason it was done that way is that the optimisation could be developed by someone who did not need to understand the logic of the code that generates the catalog files in the first place.

This was a side-effect of the fact that multiple people are involved in the development and they do not understand all the program logic at a detailed level so try to make changes/improvements that are as localised as far as possible.

That does not mean that further improvements are not possible. I am not sure how easy it might be to use a move/rename from Java and in a platform independent way. Definitely worth looking at though.

Finally I was unsure what the "Minimise number of changed files" option does.

This avoid copying to the final target any files whose content is unchanged from the previous run. In earlier releases when this optimisation did not exist, users were finding that when using systems like Dropbox many files whose content was the same as the previous run but had a newer date/time ended up being copied and thus needing to be uploaded to the 'cloud'

I had thought that if calibre2opds read the date & time of the original _catalog creation and also placed the CRC of the profile.xml used to generate it in the _catalog, then upon start it could see if the profile.xml being used is identical, and use the date & time stamp to only process books that have been amended after this date & time. In my case its seems to always process all books, and only cary out a crc check on the generated files afterwards.

Might be worth thinking a bit about that. I think at the moment the profile.xml is always written out so its date/time stamp may not be usable without some change to program logic.

It might now be possible to use the timestamp of the metadata.opf file in the Calibre library to work out when a particular book was ast amended. At the time most of the Calibre2Opds logic was developed that file did not exist.

Also for some reason my synclog.log always says "IGNORED (CRC)" for the files.

Not sure about that. That log was only added very recently so I am not sure how good it is. The ignored CRC message would normally means some other criteria (such as a changed file size) was used to decide on whether to do the copy.

[itimpi]

Originally Posted by electronicfur

Does anyone know what the following error means? I get it with about 2000 of my books, and it results in empty book*.html and xml files. I have looked in the temp directory when calibre2opds is generating and the xml files causing problems are 0 bytes, which explains the reason for the error. But I cant figure out why they are zero bytes, as this is the first WARN level message in the log. And the books in question look fine in the Calibre library.

You did not mention exactly what version of Calibre2Opds you are using? Quite a lot of fixes were included in the 2.4 release that could cause Java exceptions.

To investigate further we would probably need a copy of your Calibre metadata.db file.

[whodean]

Originally Posted by whodean

<---has been awaiting a Mac version of the beta

Any word?

[itimpi]

Originally Posted by whodean

Any word?

I'll have to email Dave Pierron again - he said he would do it.

Note that as a stop gap it should be possible to upgrade from the 2.3 release using the ZIP file as mentioned in the documentation. Having said that, I have not checked out myself how good the instructions (very brief) are as I do not have a Mac to test it on myself.

[whodean]

I couldn't make that work, itimpi

[itimpi]

Originally Posted by electronicfur

Does anyone know what the following error means? I get it with about 2000 of my books, and it results in empty book*.html and xml files. I have looked in the temp directory when calibre2opds is generating and the xml files causing problems are 0 bytes, which explains the reason for the error. But I cant figure out why they are zero bytes, as this is the first WARN level message in the log. And the books in question look fine in the Calibre library.

Just to say that I have reproduced this error on my test library. Like you I cannot see any obvious reason why the XML file is 0 bytes - but at least I ahve something to look for now.

On a slightly different note I am going to reword entries in the Sync Log. Therefore isntead of an entry reading
IGNORED (CRC)
it will now read
NOT COPIED (CRC same)
which is I hope a bit clearer. Other messages getting a similar overhaul.

[Xenophon]

itimpi --

Which version of trook do you use with the latest calbre2opds? I used the upgrade-from-2.3 via zip file approach to upgrade my calibre2opds. It seems to run fine, produces a catalog, and sends it to the nook. It also copies books as expected. Sadly, trook can't seem to do anything with the catalog -- not even open it.

I'm using the craftycoder version of trook, on a nook classic wifi running the 1.4 firmware. Any advice?

[itimpi]

Originally Posted by Xenophon

itimpi --

Which version of trook do you use with the latest calbre2opds? I used the upgrade-from-2.3 via zip file approach to upgrade my calibre2opds. It seems to run fine, produces a catalog, and sends it to the nook. It also copies books as expected. Sadly, trook can't seem to do anything with the catalog -- not even open it.

I do not have a nook (and therefore do not use trook). It is always possible that something is broken in the trook support, but if so I cannot either check that out or fix any issue. The trook functionality was developed by Dave Pierron, so hopefully he might be in a position to comment.