Content Server location is a problem

[Daria9000]

I just bought the Calibre Companion for my Kindle Fire and have set it up with my content server.

Then I realized there is a major flaw in the structure for me ... unless I am missing something.

I can only use the Calibre app if I am connected to the Content Server, right? In other words, it doesn't seem possible to use it "offline". If I am missing something, please let me know.

The issue: my content server is on my PC. This is a work computer (professional and personal). When I am working on the PC, my Fire is off and elsewhere.

When I'm using the Fire, for reading or whatever, I am not at the PC, and it is turned off. I do not want to leave the PC on 24hrs/day just so I can use Calibre on the Fire...to say nothing of the fact that it wouldn't help when I'm using the Fire away from home.

It seems to me there are 2 potential solutions: 1. having some way to store my Calibre content in the cloud. I suspect this would require more resources than the Calibre community has.

Second, create a way to download the catalogue only (authors, titles, metadata) to the Fire. Ideally, it would indicate if a book is stored locally or not (just as Amazon distinguishes between books in the cloud and books on the device).

I now have so many ebooks that Amazon's display options, including its pitiful collections, are very limited. I had hoped that Calibre on the Fire would give me the display flexibility, especially for series, genres that the desktop version offers.

But if I can't use the Companion offline, it's useless.

I don't mind having spent the money. I've gotten quite a lot of value from the desktop version. I'm simply disappointed that it's not going to work for me on the Fire ... unless there is an option I have missed.

[chaley]

Originally Posted by Daria9000

I just bought the Calibre Companion for my Kindle Fire and have set it up with my content server.

Then I realized there is a major flaw in the structure for me ... unless I am missing something.

I can only use the Calibre app if I am connected to the Content Server, right? In other words, it doesn't seem possible to use it "offline". If I am missing something, please let me know.

No, you are not right. You only need to be online if you want to fetch books from the content server. Once you have fetched a book, either from the content server or via the wireless device connection, it is on your device and can be used offline.

However, reading below it seems that your problem is that you want to access the content server when it is turned off. This is a different issue.

The issue: my content server is on my PC. This is a work computer (professional and personal). When I am working on the PC, my Fire is off and elsewhere.

When I'm using the Fire, for reading or whatever, I am not at the PC, and it is turned off. I do not want to leave the PC on 24hrs/day just so I can use Calibre on the Fire...to say nothing of the fact that it wouldn't help when I'm using the Fire away from home.

It seems to me there are 2 potential solutions: 1. having some way to store my Calibre content in the cloud. I suspect this would require more resources than the Calibre community has.

What I and many people do is to put the content server itself in the "cloud". This technique works well but it does cost money. Here is one discussion about how to set it up. Currently Linode, the company I use, charges $10/month for a server powerful enough to run calibre's content server 24/7.

Another option is to use Calibre Cloud instead of CC, but I think it does require you to be online because your library must be in Dropbox etc.

Second, create a way to download the catalogue only (authors, titles, metadata) to the Fire. Ideally, it would indicate if a book is stored locally or not (just as Amazon distinguishes between books in the cloud and books on the device).

Many people do this by simply keeping their entire library on the device in CC. I have seen people using CC with libraries in excess of 15,000 books.

With your scheme how would you get a book that wasn't on the device? I understand you to be saying that when you have your fire in hand the content server is guaranteed to be off and the computer running calibre is not available.

I now have so many ebooks that Amazon's display options, including its pitiful collections, are very limited. I had hoped that Calibre on the Fire would give me the display flexibility, especially for series, genres that the desktop version offers.

But if I can't use the Companion offline, it's useless.

I don't mind having spent the money. I've gotten quite a lot of value from the desktop version. I'm simply disappointed that it's not going to work for me on the Fire ... unless there is an option I have missed.

It seems that your library is permanently separated from your fire. There is no way to connect the fire to calibre either to a content server or as a wireless device. In this case you are correct, CC is probably useless.

[Daria9000]

Just trying to clarify here.
1. I don't need Calibre to transfer books either to or from the Fire HDX. 99% of the e-books I have are stored in Amazon's cloud and I download to whichever Fire device I want to read the book on.

2. I got the desktop version of Calibre primarily to deal with the 1% of books I get from other sources and, most especially, to get them into a mobi format which will turn up as a book rather than a DOC in the Fire.

3. It doesn't make sense to me to use CC to manage the library on the device. I have multiple FIREs and switch between two of them regularly so Amazon's cloud is my point of contact and is available wherever I have wifi.

4. I have duplicated my Amazon library on my desktop PC, and keep it more or less up-to-date, and have slowly been creating tags so it is easier for me to know exactly what I own. So, in a way they are mirrored.

5. I had been hoping that what I think of as the catalogue, as opposed to the books themselves, was more-or-less exportable.

6. I always have wireless access, simply not to the PC on which my Calibre Content Server exists.

7. What I want on my Fires is the metadata, the tags that Calibre has. I'm not sure how Dropbox would help. It would simply be another place where my books are stored, right? So I could upload or download them - which I don't need because almost all my Kindle books are stored in Amazon's cloud.

So, long story short, I am correct: there is no way to access the metadata via Calibre Companion unless I'm connected to the content server?

[chaley]

As I said, you can access metadata for books that have already been downloaded. If I were you I would download my entire calibre library to CC and be done with it. This is especially true given that we are talking about 1% of the books.

I understand that you are saying that you don't want to use calibre's mail-to-amazon function because the books will end up as personal documents and not as books. In any event, CC wouldn't be involved here.

I understand that you are saying that you want to access the metadata in a calibre library for books not on your device while either or both of the fire and the content server are offline. CC cannot do this. CC can access metadata for books in your calibre library but not on your device only if your Fire and either a calibre content server or a calibre GUI instance (wireless device) are online. There is no way for CC to store a calibre library's metadata offline without storing the books as well.

My take: the conditions you have set make CC not the right product for you. I would offer a refund but Amazon makes that very hard to do. My apologies.

[Daria9000]

Clarification.
1. I've gotten a lot of use out of calibre so don't mind the money I spent on the Companion even if it turns out not to be useful at this point.

2. I want to access calibre information for books that are ON my device. But I don't want to be connected to the Content Server in order to do that.

Let me approach this a different way.

For me, Calibre serves several functions.

First, it lets me convert non-Amazon books into a Kindle format so I can download them to my Fire and find them in the Books section. From my perspective, doing this with a USB connection is the most practical option since what normally happens is that I have downloaded a book from another source directly to my Fire. So I want to get it into Calibre from my Fire, change the metadata, convert it to mobi, copy it back to my Fire. The USB connection works fine for this function.

Second, and this is something I've only been playing with for a short while, Calibre lets me add tags which give me more flexibility in how I look at my e-books. I see it as the digital equivalent of my physical bookshelves (some house sci-fi, some fantasy, some mysteries, another for novels, one for business books, etc.) and within those bookshelves I sort books by author and, within author, by series (when they exist).

Third, what I find awful about Amazon's handling of books is how limiting it is. I can sort books by author, title, recent. Nothing else. I can create collections, but I can't even sort books within those collections.

So, basically, Calibre does what Amazon doesn't do and if I read books on the computer, well, that would be that.

But, right now I have multiple book databases: a complete one is in Amazon's cloud, incomplete ones are stored locally on my various Fires, and a complete one (the "physical" books) exists on my PC in the Calibre database. (The PC Calibre database is essentially a duplicate of the Amazon cloud.)

If, from the Fire, I connect to the Content Server, I can see the various groupings that I've set up so far, can sort books in various ways. But when I disconnect from the Content Server, the Calibre Companion doesn't show anything on the Fire, doesn't see any of the books that are on my Fire. I go to Groups and there just isn't anything there.
That's the part I don't understand.

Content Server on PC: all the ebooks I own.
Books on Fire1 or Fire2 : subset of all the ebooks I own.

So this is how I thought the Companion would act or how I would like it to act.
1. Periodically connect to content server to get/update all the data (not download the physical books but download the catalog, the metadata).
2. Disconnect from content server.
3. Go into Companion, ask it to show me, say, all the books I've tagged as history books and sort by author within that group.
4. Select one of the history books.
5. Have it tell me that that book is either on my Fire or is not on my Fire.
6. Read book.

[Daria9000]

Maybe this will explain it better.

I'm on the Fire. I connect to the Content Server. I select a group of authors, select an author. To the right there are boxes (on device) that are empty. I check. The books in question ARE on my device. In fact, the Companion tells me that I have NO books on my device - which is most definitely not true.

So do I have to download a book from the Content Server to the Fire for the Companion to see it? In other words, do I have to have two copies of the book, one for Amazon and one for the Companion?

[chaley]

Originally Posted by Daria9000

So do I have to download a book from the Content Server to the Fire for the Companion to see it? In other words, do I have to have two copies of the book, one for Amazon and one for the Companion?

Yes and no.

First, the fundamentals behind the "yes". CC does not store metadata for books that are not on the device, where "on the device" means downloaded by CC. When connected to the content server, CC shows information in calibre's library. When disconnected, CC shows information in the copy of the calibre library that has been downloaded to CC. So yes, CC must download something from calibre to the device for CC to "see" it when not connected. CC cannot see books on the device that it didn't download, such as books you downloaded using Amazon's library manager.

The "no" is there because Amazon's reader app can see CC's copy. It does not require a second copy.

So this is how I thought the Companion would act or how I would like it to act.
1. Periodically connect to content server to get/update all the data (not download the physical books but download the catalog, the metadata).
2. Disconnect from content server.
3. Go into Companion, ask it to show me, say, all the books I've tagged as history books and sort by author within that group.
4. Select one of the history books.
5. Have it tell me that that book is either on my Fire or is not on my Fire.
6. Read book.

CC cannot do step 1, download a catalog but not the books, which means it cannot do step 5. Either the book is in CC's copy of calibre's library in which case you can see and read the book, or the book is not in CC's copy in which case you can't do any of the steps 3-6.

I see two solutions:

1) The best solution is the one I mentioned earlier and one used by many CC users: put your entire calibre library on your Fire(s). From time to time you would connect your Fire to calibre as a wireless device to update metadata for existing books and get copies of new books (steps 1 and 2). By default the Amazon version of CC stores the books in the folder Amazon's reader uses (IIRC /Books) so the Amazon reader can see them and possibly sync reading positions, annotations, etc., without requiring a second copy. You would not use Amazon's library manager, nor would you use Amazon's cloud except for anything the reader app does with it. The upside of this solution is that CC has visibility to the real library and maintenance is easily done by connecting as a wireless device from time to time. The downside is that you must have enough space on the fire to store all the books. This solution takes care of all the steps above, except that the "not on the fire" part of step 5 never happens.

2) You could use CC as a catalog, manually integrating it with Amazon's library manager. Another user had a similar use case. Her library consisted mostly of physical books. She wanted the metadata in CC, IIRC to check if she already had a book to avoid buying another copy. We solved her problem by creating an empty txt format for every book in her calibre library, setting up CC to only download txt formats, then downloading every book from calibre to CC (just as in solution 1 above). In this way she ended up with CC fully populated with her books but with minimal space used. From time to time she connects to calibre as a wireless device to update metadata and get new books. Here is the thread discussing this solution. Here is another thread discussing something similar.

Under this second scheme you would need to note the title and author of the desired book then go to Amazon's library manager to see if the book is on the device. If it is not then you would fetch it from Amazon's cloud. This is the manual integration part. CC takes care of steps 1 through 3 and part of 4, with the note that the periodic connection is best done using the wireless device so that metadata in CC is updated for books already there. CC cannot handle steps 5 and 6 because the actual books are not in CC and CC cannot see books downloaded from Amazon's cloud.

[DoctorOhh]

I've never used the content server function with CC. If I had a clean tablet and installed CC then only ever used the content server to download books would CC display the metadata for those books? At one point I thought you needed to connect as a wireless device to update the metadata, I'm guessing this is not the case.

[chaley]

Originally Posted by DoctorOhh

I've never used the content server function with CC. If I had a clean tablet and installed CC then only ever used the content server to download books would CC display the metadata for those books? At one point I thought you needed to connect as a wireless device to update the metadata, I'm guessing this is not the case.

Books downloaded using the content server come with their calibre metadata. Embedded metadata (if any) is ignored. This is the main reason that CC requires a calibre content server and not a plain OPDS server. However, the content server does not update metadata unless the book is sent again.

One advantage of the wireless device connection beyond having the power of calibre's UI while you are connected is that metadata is automatically updated for books in CC without needing to send the books again. Another is the sync functions. Another is the power of calibre's plugins such as reading list and smart eject.

We originally intended that the wireless device be the "go to" connection, and the content server was for occasional use. Of course we were immediately surprised that some people use only the content server. I am not sure why other than possibly having a 24/7 content server somewhere, meaning that one does not need to launch calibre or its computer.

[DoctorOhh]

Originally Posted by chaley

Books downloaded using the content server come with their calibre metadata. Embedded metadata (if any) is ignored. This is the main reason that CC requires a calibre content server and not a plain OPDS server. However, the content server does not update metadata unless the book is sent again.

One advantage of the wireless device connection beyond having the power of calibre's UI while you are connected is that metadata is automatically updated for books in CC without needing to send the books again. Another is the sync functions. Another is the power of calibre's plugins such as reading list and smart eject.

We originally intended that the wireless device be the "go to" connection, and the content server was for occasional use. Of course we were immediately surprised that some people use only the content server. I am not sure why other than possibly having a 24/7 content server somewhere, meaning that one does not need to launch calibre or its computer.

Thanks for the clarification I had a brain spasm that refused to go away until I asked.

[CRussel]

Originally Posted by chaley

We originally intended that the wireless device be the "go to" connection, and the content server was for occasional use. Of course we were immediately surprised that some people use only the content server. I am not sure why other than possibly having a 24/7 content server somewhere, meaning that one does not need to launch calibre or its computer.

Easy to explain -- I have little need for connecting as a wireless device -- I can plug in a cable from any of my PCs that run Calibre and transfer that way. BUT, I'm reading a book, discover that I don't have the sequel on my device, and I don't want to get out of bed, fire up my PC, and download the book? NOW that content server that sits on my Windows Storage Server Essentials is suddenly my go-to option. (If one could PULL content from a Calibre repository without having to run the content server, that would be even better, but I can live with this.)

My only real complaint? CC doesn't handle dual NIC servers well. (I have two networks in my house -- my domain network and a separate network that connects to a different wireless host and is only used for entertainment stuff. The Windows Storage Server Essentials box has two NICs and sits on both networks. CC didn't like that, and I finally had to force it to only use one network. I usually have to switch wireless networks to connect to it.

[chaley]

Originally Posted by CRussel

My only real complaint? CC doesn't handle dual NIC servers well. (I have two networks in my house -- my domain network and a separate network that connects to a different wireless host and is only used for entertainment stuff. The Windows Storage Server Essentials box has two NICs and sits on both networks. CC didn't like that, and I finally had to force it to only use one network. I usually have to switch wireless networks to connect to it.

Assuming that your server has two fixed IP addresses, doesn't entering one of them into each of the two sections of CC's Content server IP settings box solve the problem?

I can imagine why using automatic connection sometimes won't work. The server will broadcast one of its IP address using mDNS (bonjour) and CC will attempt to connect to that address. It is apparently broadcasting the one you normally don't use. You could fix this by telling the content server to use the NIC on the network you usually connect to by using the calibre tweak (What interfaces should the content server listen on (ID: server_listen_on)). Of course that will ensure that you can never connect on the other NIC.

[Daria9000]

Thank you.

In short, to use the Companion as it has been designed, one must, essentially, substitute Calibre for Amazon.

For me, there are several problems. First, much as I like Calibre, I'm not prepared to give it sole responsibility for my ebooks. I am, for better or worse, wedded to Amazon at this point. I buy e-books almost exclusively from Amazon and download them regularly to my Fire. As I understand your explanation, every time I downloaded an Amazon book, I'd have to send it to Calibre, incorporate it in the Calibre library, download it from Calibre and then delete the Amazon copy...because I found last night that the folder setting in the Calibre Companion options on both of my Fire HDXs is empty. Amazon does not store its books in that location.

Second, since I have multiple Fires, with different amounts of storage. I don't particularly want to download my entire ebook library (which now consists of about 1000 ebooks) to each device. That's the whole point of Amazon's cloud: the ability to upload and download books regardless of the device one is using.

Third, and related to the first point, I have had problems in the past with Amazon's ability to sync book notes and highlights across devices. Introducing Calibre Companion, as currently designed, into the mix would pretty much make it impossible for me to get any help from Amazon when/if more problems occur in the future.

I'll take a look at the text option and see if that might be useful.

I do have a suggestion: I think the description of the Calibre Companion app should emphasize that it must be used as a replacement for Amazon's library management not as an addition to or enhancement of Amazon's library.

Lastly, I want to thank you very much for taking the time to explain everything to me.

[CRussel]

Originally Posted by chaley

Assuming that your server has two fixed IP addresses, doesn't entering one of them into each of the two sections of CC's Content server IP settings box solve the problem?

I can imagine why using automatic connection sometimes won't work. The server will broadcast one of its IP address using mDNS (bonjour) and CC will attempt to connect to that address. It is apparently broadcasting the one you normally don't use. You could fix this by telling the content server to use the NIC on the network you usually connect to by using the calibre tweak (What interfaces should the content server listen on (ID: server_listen_on)). Of course that will ensure that you can never connect on the other NIC.

I ended up forcing Calibre to only use the one NIC. It worked, but means I often have to switch wireless networks to connect. Annoying but not any worse than that.

[DoctorOhh]

Originally Posted by Daria9000

I do have a suggestion: I think the description of the Calibre Companion app should emphasize that it must be used as a replacement for Amazon's library management not as an addition to or enhancement of Amazon's library.

Interesting suggestion. Calibre Companion simply is a direct companion for books folks maintain in their calibre library. I could be wrong and I don't have a Fire, but I don't see any reason to think the two apps can't be used side by side without interfering with each other.