Managing a huge collection of PDF files

[pmh]

Hi all,

new to the forum, using Calibre for a couple of years and really enjoying the experience. I manage my library of ebooks (ePub format) and sync with a pocketbook reader. So far so good.

My question is which approach to take - or what you are doing - for a large collection of PDF documents that do not really fit the "book" metaphor well - scientific articles, magazines, manuals ... accumulated over more than two decades. (I'm old )

Besides the metadata that does not fit a "publisher, author, title" format, many of those documents are formatted for a fixed page size like letter or A4, also frequently colour, and are not readable on a handy s/w e-ink device like my pocketbook.

I'm pondering if I should still go ahead and throw them all in Calibre and limit syncing to ePub. Then possibly use the filtering features.

Or just use folders and the Mac OS builtin spotlight.

Or is there anything else that do not yet know? Hence this post.


Thanks in advance for comments, kind regards,
Patrick

[DNSB]

I have a few items like that which live in their own libraries in calibre. Mostly tech manuals left over from before my retirement and household appliance manuals. Quite a few custom columns that are not needed for the main library. The other library is scans of old SF & F magazines. Oddly, those two libraries have a lot fewer books than my main library but eat disk space like it was free.

[theducks]

My custom columns are general, so I just keep my Manuals in my main Library.
Except for Author and Title, I don't believe the Police will seek you out for leaving fields blank.

In case of Manuals: if the author is not specified, I use the Publisher there also (sometimes embellished ). eg LG Staff
I also tag the document: UG (users Guide), QSG (Quick Start Guide), IG (Installation Guide), Wnty ...

I even use 'Series' to tie a set together (LG BD [0])

[DrNefario]

I add my PDFs to Calibre. I use the content server to transfer them to my iPad for reading. I made a separate Library for comics, in the end, since they were getting in the way, but the rest of my miscellaneous PDFs - game rules, magazines, etc - are just in the main library.

[ownedbycats]

Originally Posted by theducks

My custom columns are general, so I just keep my Manuals in my main Library.
Except for Author and Title, I don't believe the Police will seek you out for leaving fields blank.

In case of Manuals: if the author is not specified, I use the Publisher there also (sometimes embellished ). eg LG Staff
I also tag the document: UG (users Guide), QSG (Quick Start Guide), IG (Installation Guide), Wnty ...

I even use 'Series' to tie a set together (LG BD [0])

I do this.





The pub-type column is shared for magazines and manuals. Product is for manuals only. I have VLs and view manager views for both.

Originally Posted by DNSB

I have a few items like that which live in their own libraries in calibre. Mostly tech manuals left over from before my retirement and household appliance manuals. Quite a few custom columns that are not needed for the main library. The other library is scans of old SF & F magazines. Oddly, those two libraries have a lot fewer books than my main library but eat disk space like it was free.

In my library, I think the PDFs of scannned fanzines and strategy guides are twice the size of everything else.

[salamanderjuice]

Originally Posted by pmh

Hi all,

new to the forum, using Calibre for a couple of years and really enjoying the experience. I manage my library of ebooks (ePub format) and sync with a pocketbook reader. So far so good.

My question is which approach to take - or what you are doing - for a large collection of PDF documents that do not really fit the "book" metaphor well - scientific articles, magazines, manuals ... accumulated over more than two decades. (I'm old )

Besides the metadata that does not fit a "publisher, author, title" format, many of those documents are formatted for a fixed page size like letter or A4, also frequently colour, and are not readable on a handy s/w e-ink device like my pocketbook.

I'm pondering if I should still go ahead and throw them all in Calibre and limit syncing to ePub. Then possibly use the filtering features.

Or just use folders and the Mac OS builtin spotlight.

Or is there anything else that do not yet know? Hence this post.


Thanks in advance for comments, kind regards,
Patrick

For scientific articles/things with DOIs I use a self hosted i librarian instance (https://github.com/mkucej/i-librarian-free).

Works well for citation management, has full text search and works well across devices.

[patrik]

Since you are familiar with Calibre, having a second library for those PDFs seems to be an easy choice (it's really easy to have several libraries controlled by the "same" Calibre, in case you haven't tried it yet). Using shelves/categories you can tag them as you please.

[pmh]

Thank you very much for all your comments.

So that's two tasks for me now it seems:

- learn how to use more than one library with Calibre and give that a spin
- checkout that I librarian thing

Honestly I do not actually *use* Calibre much. Its main purpose for me is storage of e-books and pushing them onto my device.

For about everything I can I prefer self hosted web applications. The desktop is dead, the browser is the new desktop. But you cannot sync your device from CalibreWeb ...

I am very proficient in self hosting all sorts of things, like Nextcloud, Paperless-NGX, Home Assistant ... and run a small home lab with two TrueNAS servers and an OPNsense - so maybe I librarian is the way to go.

Or use Calibre for management and sync the libraries to my Nextcloud or similar and then put a CalibreWeb in front of that for access from anywhere.

Thanks again, take care,
Patrick

[jackm8]

May I ask why aren't you just using file folders to organise your collection, if you're not that keen on Calibre? That's what I do with my photos. I put them each into an individual folder, and write a few vague tags to the folder's name. It's the only way not to get into library management nightmares ragarding Capture One and Lightroom, when it comes to transferring libraries from one to the other.

[theducks]

Originally Posted by pmh

Thank you very much for all your comments.

So that's two tasks for me now it seems:

- learn how to use more than one library with Calibre and give that a spin
- checkout that I librarian thing

Honestly I do not actually *use* Calibre much. Its main purpose for me is storage of e-books and pushing them onto my device.

For about everything I can I prefer self hosted web applications. The desktop is dead, the browser is the new desktop. But you cannot sync your device from CalibreWeb ...

I am very proficient in self hosting all sorts of things, like Nextcloud, Paperless-NGX, Home Assistant ... and run a small home lab with two TrueNAS servers and an OPNsense - so maybe I librarian is the way to go.

Or use Calibre for management and sync the libraries to my Nextcloud or similar and then put a CalibreWeb in front of that for access from anywhere.

Thanks again, take care,
Patrick

Adding (and using) a separate Library is easy

Right click the Library Icon in Calibre Switch/Create: In new Location Browse to where you you want to put it (Never inside another Library): Create an empty Library The tick Copy Structure is for when you want the new one to inherit the same customizations as the current (eg. a kid friendly one where a VL is just too risky from little eyes)

Calibre switches to the new one when successful. To switch back (and forth), the other libraries will appear on the quick switch List Be sure to give each a unique name to make life easy)

That is it.
Filling in the metadata is the hard part

[JSWolf]

Originally Posted by pmh

Hi all,

new to the forum, using Calibre for a couple of years and really enjoying the experience. I manage my library of ebooks (ePub format) and sync with a pocketbook reader. So far so good.

My question is which approach to take - or what you are doing - for a large collection of PDF documents that do not really fit the "book" metaphor well - scientific articles, magazines, manuals ... accumulated over more than two decades. (I'm old )

Besides the metadata that does not fit a "publisher, author, title" format, many of those documents are formatted for a fixed page size like letter or A4, also frequently colour, and are not readable on a handy s/w e-ink device like my pocketbook.

I'm pondering if I should still go ahead and throw them all in Calibre and limit syncing to ePub. Then possibly use the filtering features.

Or just use folders and the Mac OS builtin spotlight.

Or is there anything else that do not yet know? Hence this post.


Thanks in advance for comments, kind regards,
Patrick

If you actually want to read these PDF, I suggest you get a tablet to use to read them. Your Pocketbook is going to be awful trying to read those PDF and converting to ePub is going to be a mess.

[pmh]

Originally Posted by jackm8

May I ask why aren't you just using file folders to organise your collection, if you're not that keen on Calibre?

I wrote in my initial post:

Originally Posted by pmh

Or just use folders and the Mac OS builtin spotlight.

Originally Posted by JSWolf

If you actually want to read these PDF, I suggest you get a tablet to use to read them. Your Pocketbook is going to be awful trying to read those PDF and converting to ePub is going to be a mess.

I also wrote in my initial post:

Originally Posted by pmh

[...], many of those documents are formatted for a fixed page size like letter or A4, also frequently colour, and are not readable on a handy s/w e-ink device like my pocketbook.

That "s/w" should be "b/w", of course. My native language being German got the better of me.


I am really just looking for opinions and info what others are doing with "papers" and "ezines". I am just starting to actually organize the cruft that accumulated over the years and which is in no order at all at the moment - scattered across project or customer related folders etc.

Also I intend to systematically download the full backlog of e.g. Journal of Tolkien Research, Mythlore, FreeBSD Journal etc. - all freely available online, but I do not trust online resources and I want to have this stuff somewhere on a local ZFS pool. And then a backup copy on two more ZFS pools ...


For another example - yesterday I found this presentation:
https://www.isc.org/docs/2023kea_custom_options.pdf

I need this now for something I am working on. I also want a place for this where I can find it in ten years should I ever need it again. Not a bookmark. Local storage. That's the point. And I am completely open to *anything*.

Thanks again for all suggestions.

Kind regards,
Patrick

[pmh]

So thinking ...

- Calibre Web can only use one library at a time
- custom metadata fields like @ownedbycats uses are easy
- so add a "book type" or similar category field instead of a second library, only sync proper "books" to the e-reader
- have everything available in a folder structure in Nextcloud (automatically, Calibre creates a folder structure - by Genre/Author sort/Title in my installation)
- also have everything available on a self hosted web app for easy tablet access
- ...

Looks like a plan.

P.S.

There's a ready to use FreeBSD port for it: https://www.freshports.org/www/py-calibreweb/
I consider that a sign

[ownedbycats]

Are you planning to run Calibre llibrary on cloud? Not best idea.

[pmh]

I run Calibre on my Macbook Pro. That is the single source of truth.

I already sync that folder to my self hosted Nextcloud, just so I can get the books in case I need them and I am not at my Mac. No fancy UI here, just the folder structure Calibre creates automatically available through Nextcloud.

Now I will probably add another FreeBSD jail running Calibre Web with access to that very same library (Nextcloud runs on my TrueNAS server and Calibre Web will, too). So I can have a more convenient web based access to the library, e.g. on mobile devices.

And then I will pour all those PDFs into that library.

That is the plan so far - any objections or serious caveats?

Kind regards,
Patrick