Calibre2opds v3.2 - OPDS catalogs from your Calibre library

[ki5ck]

Yes, I use Marvin now. Before the iPhone release, I used Shubook.
Thanks for your help.

[slantybard]

Using Mac OS X Mountain Lion, the installation of the zip RC3 did not work for me. I followed the instructions at https://wiki.mobileread.com/wiki/Calibre2Opds_Installing for the Mac and copied every file from the zipped folder into my Java folder inside the COPDS package on my Mac. I then tried to change the rungui.sh using terminal, however, when I double-click on the rungui.sh now, it opens up in xcode, not as an executable shell. Also, when I open the "app" it shows up as version 3.1 rev 170.

Not sure where I screwed up...

[itimpi]

Originally Posted by slantybard

Using Mac OS X Mountain Lion, the installation of the zip RC3 did not work for me. I followed the instructions at https://wiki.mobileread.com/wiki/Calibre2Opds_Installing for the Mac and copied every file from the zipped folder into my Java folder inside the COPDS package on my Mac. I then tried to change the rungui.sh using terminal, however, when I double-click on the rungui.sh now, it opens up in xcode, not as an executable shell. Also, when I open the "app" it shows up as version 3.1 rev 170.

Not sure where I screwed up...

I must admit I do not have a Mac myself so not sure where you have gone wrong as I have never tried those instructions myself (they were added by a user).

I will spend some time today putting together a .dmg file that can be used instead (using tools on a Windows system that I have for this) and make that available for download which will hopefully work better for you.

[slantybard]

Thanks itimpi, I appreciate it.

[cyana]

Hello itimpi,

I'd have an urgent feature request - don't know if this is possible, though, however I think lot of people using calibre2opds for a cloud ebook storage will run into this problem:

There are too many folders generated over the time. I have about 2,500 ebooks resulting into around 1,000 authors. Updating is not possible anymore because I get constant errors from the cloud service that I've exceeded the storage limit. In fact I have only exceeded the maximum numbers of folders possible.

Even when you use a cloud service that has no such limit, updating lasts hours because the folder structure is read over and over again. So my suggestion would be to insert a layer between the main folder and the authors, such as literals or something like that, e.g.

eBooks
----A
------Alice Bob
------ ...
----B
----C

.... and so on.

This would cut down the problem dramatically, because each literal folder would contain maybe 20 to 100 subfolders, that are read within a blink of an eye.

Another cool feature would be a link to the author page where ever the name of the author appears.

And finally - it would be cool to have an option in between yes and no for 'Do not generate cross-reference links'. I want cross reference links, however they don't need to reflect the current state. It does not matter if there are 789 Thrillers or 792. This would avoid that every single book page is updated again when you add new books.

thanks!
Cyana

[itimpi]

I am not sure if you are talking about the folder/file structure used within the Calibre library, or that used in the Calibre2opds generated catalog? The structure of the Calibre library is mandated by Calibre and not something that Calibre2opds can change.

Have you tried the recently posted 3.2 release of calibre2opds? You will find it generates a lot less folders than the previous release did, and the folder/filenames are more understandable as to their purpose. However the total number of files does not really change as that is determined by the number of pages required to describe your library. Also if you do not want both the HTML and OPDS versions of the catalog then only generating one of them halves the number of files generated.

The program already has code in it to avoid re-reading file information, so I am not sure why you think it is continually re-reading them. Also if you are using a Cloud service have you set the option (on the Advanced tab) to Minimize Number of Changed Files. That can dramatically reduce the number of files that will need syncing to the Cloud on each run.

[cyana]

Hello itimpi,

thanks for your swift reply! Yes, I was talking about the Calibre-folder structure. I was hoping that it is not mandatory for the generated web site of Calibre2opds (it's necessary for the generation, of course - but I was talking about the output).

I've updated to 3.2 today - seems I should also update the German localization . So in my previous post I was referring to 3.1, where all book pages are re-generated again every time you update/upload. I will give the minimize option a try.

thank you
cyana

[itimpi]

I am afraid the calibre structure is not variable. Calibre2opds catalogs merely act as pointers into it.

The Minimize Changes option was also available in 3.1 (in fact I think it came in at 3.0). All pages are ALWAYS currently generated in a temp folder. The Minimise Changes avoids copying any that are identical to last time to the actual live catalog area which is particularly important for catalogs that need syncing to the Cloud.

[cyana]

Is it maybe possible to put the _catalog folder into another place?

As for the minimize option, I understand now that calibre2opds generates all files first but then makes a selection which ones to upload. In my case, I have to use a synchronizer specific to the cloud, that synchronizes all new files or files with a newer generation date than the file in the cloud, that's why I come across this problem. Unfortunately the WebDAV interface does not work properly with my cloud service, they say it's a windows problem.

Can you recommend a cloud service that works?

thank you
cyana

[itimpi]

Originally Posted by cyana

Is it maybe possible to put the _catalog folder into another place?

At the moment this is not possible. I have added it as a a feature request at http://calibre2opds.myjetbrains.com/.../issue/c2o-160. It may well make it into the 3.3 release as it should not be hard to implement. The main difficulty is likely to get a clear view of how it should work from a user perspective. If you have any thoughts they could be added as comments to this issue.

As for the minimize option, I understand now that calibre2opds generates all files first but then makes a selection which ones to upload. In my case, I have to use a synchronizer specific to the cloud, that synchronizes all new files or files with a newer generation date than the file in the cloud, that's why I come across this problem. Unfortunately the WebDAV interface does not work properly with my cloud service, they say it's a windows problem.

Are you saying that you do not have a local TEMP folder that can be used as working space while calibre2opds is running? If you do then as long as you have set the Minimize Changes option you should not hit this problem. There is actually a feature request (http://calibre2opds.myjetbrains.com/.../issue/c2o-158) that may make this setting irrelevant if it can be implemented (but there is no guarantee that it can).

Can you recommend a cloud service that works?

It will depend on exactly what your requirements are. I currently use Dropbox and Copy.com successfully. I have experimented with others but do not like them as much. I am currently trying MegaCloud as well but experience occasional sync issues with that service so I am not sure I will keep it installed. Others I have tried and abandoned as they do not meet my requirements include GoogleDrive, SkyDrive, Box.net and SugarSync. New ones keep appearing so that list is by no means exhaustive as to the possibilities.

[ki5ck]

Originally Posted by cyana

Can you recommend a cloud service that works?

I use Dropbox, and I have over 3800 books. No problems for me.

[cyana]

Thanks itimpi and ki5ck for your replies.

Not sure about the temp folder. I'm generating the catalog directly in the sync folder, and these files regularly update their creation-date after each generation.

At the moment, I'm in contact with the support people of my cloud service - let's see how it turns out. If it does not work I'll drop them

best greetings & thanks
cyana

[itimpi]

Calibre2opds does not ever generate directly to the target folder. It generates to a folder under the system TEMP location and after doing that copies the results to the final location. If the option to Minimize Changes is set then it will not copy files that are have identical contents to the previous run.

[cyana]

I just found out by accident, that the WebDAV interface works fine on winXP. So amazingly, when I use virtual XP from within my win7x64 environment, I can upload with calibre2opds. The problem that you cannot mount a webdrive to a local drive in XP can be solved with NetDrive.

So I just want to report a bug I came across - when adding custom catalogs, the last line is not saved and will be ignored when you generate the catalogs. So you must enter it twice, save, close and reopen. The previous last line then has disappeared, the contemporary last line however is kept. This already applied also to the previous versions of 3.x

As for 'Minimize Changes' - unfortunately, mostly every book page is changed when you generate the catalogs because of the cross-reference links. This is why I would suggest a 'reduced' option for cross-reference links. When a book has the tag 'Thriller', it would be enough to mention and link the tag without the always increasing number of thrillers that are in the database; it does not make a real difference if there are 812 or 813 thrillers, however it would avoid that 812 book pages are generated again when you add another thriller to the database.

best greetings and thanks for your great work!
cyana

[itimpi]

Originally Posted by cyana

So I just want to report a bug I came across - when adding custom catalogs, the last line is not saved and will be ignored when you generate the catalogs. So you must enter it twice, save, close and reopen. The previous last line then has disappeared, the contemporary last line however is kept. This already applied also to the previous versions of 3.x

The code around saving those settings has not changed since 3.1 so if there was a bug in that area in 3.1 it would also be present in 3.2.

As for 'Minimize Changes' - unfortunately, mostly every book page is changed when you generate the catalogs because of the cross-reference links. This is why I would suggest a 'reduced' option for cross-reference links. When a book has the tag 'Thriller', it would be enough to mention and link the tag without the always increasing number of thrillers that are in the database; it does not make a real difference if there are 812 or 813 thrillers, however it would avoid that 812 book pages are generated again when you add another thriller to the database.

You are right - it will make sense to omit the counts when the option to Minimize Changes is set. I also should probably remove the Do Not Save Bandwidth setting as although it controls a different internal mechanism to Minimize Changes I cannot see why they can not both be controlled by the same setting. The fact there are two settings is historical from the way the code evolved.