Managing a huge collection of PDF files

[jackm8]

Why such adherence from cloud storage? I know of many people that store their documents also on sites like Internet Archive. Not because it's safer, but because it's another backup. Chances that NAS can fail are about as high as someone sending a pull request to Internet Archive for whatever horded material you've put on it.

I mean, even pdf document that you posted as an example, contains internet links that may or may not be available in the future. By uploading it to Archives you get to access it, and it may even help people in the future that require similar information.

[ownedbycats]

Originally Posted by jackm8

Why such adherence from cloud storage? I know of many people that store their documents also on sites like Internet Archive. Not because it's safer, but because it's another backup. Chances that NAS can fail are about as high as someone sending a pull request to Internet Archive for whatever horded material you've put on it.

I mean, even pdf document that you posted as an example, contains internet links that may or may not be available in the future. By uploading it to Archives you get to access it, and it may even help people in the future that require similar information.

Calibre uses SQLite. SQLite and cloud storage/NAS don't mix.

Also a lot of people keep copyright stuff in their library. Not uploade that to IA.

[pmh]

Originally Posted by jackm8

Why such adherence from cloud storage? I know of many people that store their documents also on sites like Internet Archive. Not because it's safer, but because it's another backup. Chances that NAS can fail are about as high as someone sending a pull request to Internet Archive for whatever horded material you've put on it.

I mean, even pdf document that you posted as an example, contains internet links that may or may not be available in the future. By uploading it to Archives you get to access it, and it may even help people in the future that require similar information.

I honestly wasn't aware you could just store your private stuff on archive.org. I assumed some team was "archiving the Internet" - whatever that means.

So who guarantees archive.org will be around ten years from now? Musk and Trump won't nuke it by some means?

I have three ZFS based storage systems in 2 separate locations 15 km apart. All of this happens by the magic of ZFS snapshots and replication without any action on my part.

I do this stuff for a living, managing roughly a hundred hosts with way over a thousand customer containers aka FreeBSD jails in three different data centres.

Technology and the management thereof is not my issue. Management of my personal archive so I can find (read: even know it's there!) stuff is.

Kind regards,
Patrick

[pmh]

Originally Posted by ownedbycats

Calibre uses SQLite. SQLite and cloud storage/NAS don't mix.

I am not intending to use Calibre over a file sharing protocol.

Main installation: Mac OS, local storage. Once the database and the media files are closed, the Nextcloud client syncs that state to Nextcloud. Which is hosted on my TrueNAS, yes. But it's just a full Nextcloud installation in a FreeBSD jail.

Now installing Calibre Web would happen in another FreeBSD jail, but - and this is important - to access the library from that one no file sharing is used but a local loopback mount of one directory from one jail into another one. Local storage, full POSIX semantics.

The only issue that could arise is Nextcloud somehow messing up the bidirectional sync. If the database ever gets corrupted in any way I have hourly and daily snapshots in at least two locations with an archive time of half a year at the moment, probably will increase to one year on the "daily" backup store.

Kind regards,
Patrick

[jackm8]

eh, nevermind

[pmh]

Did I write anything to annoy you?

[theducks]

Originally Posted by jackm8

eh, nevermind

Calibre is a Hybrid database, thus the pieces need to be in lockstep ALWAYS.
By Hybrid I mean: the Metadata is in SQLlite, but the books are in the filesystem All relative to wher the DB is kept (AKA the Library folder). Calibre keeps the WHERE link in the DB. Tamper with the file name with the OS and the linkage gets broken.

[pmh]

If it's all relative to the DB root folder, I guess that is fine. Calibre Web would not work otherwise, I guess.

[meeera]

Originally Posted by pmh

My question is which approach to take - or what you are doing - for a large collection of PDF documents that do not really fit the "book" metaphor well - scientific articles, magazines, manuals ... accumulated over more than two decades.[...]

Or just use folders and the Mac OS builtin spotlight.

Or is there anything else that do not yet know?

Have you looked at Devonthink? It was pretty great when I was managing a bunch of academic articles/PDFs/etc. Looks like it's still being maintained and updated. It's pricey but powerful - depends on your needs.

https://www.devontechnologies.com/apps/devonthink

[pmh]

@meeera That looks intriguing, thanks. The price tag of the standard edition is perfectly in my ballpark. And not only don't I need any of the other features, I explicitly do not want any software that forces "AI" on me and stuff

I am only a bit concerned about the closed eco system, so still no final decision.

Tended to lean towards Calibre. But maybe since the PDFs won't go on my reader, anyway ...

I just fear I will clutter my library and the carefully created Genre tags etc. if I throw dozens of times more PDFs in there than I have books.

I guess I'll have to make the effort to install Calibre Web and then compare it with that other self hosted "I Librarian" solution.

Thanks for the suggestion.

[ownedbycats]

Originally Posted by pmh

I just fear I will clutter my library and the carefully created Genre tags etc. if I throw dozens of times more PDFs in there than I have books.

Here's how I do it.



Under the types is genres - science fiction, reference books, etc. Subjects/topics (horses, Atlantic Canada, psychology, the whole kitchen sink ) is shared between every book type.

You can also make a seperate library, use that to sort the tags, then move to the main.

[pmh]

Yeah, I already have hierarchical genres, so I could for a brute force attempt at first create a "PDF" or "papers" genre and find out how far that gets me.

I still like a Calibre approach, because I am confident Nextcloud and Calibre Web will make a one-stop solution for all my needs.

Thanks!

[meeera]

Originally Posted by pmh

@meeera That looks intriguing, thanks. The price tag of the standard edition is perfectly in my ballpark. And not only don't I need any of the other features, I explicitly do not want any software that forces "AI" on me and stuff .

Not all AI is bad. Devonthink integrated the "good kind" of AI long before consumer generative AI & LLMs were a thing. It's under your control and isn't harvesting your data. https://www.devontechnologies.com/ap...nk/security#ai

[pmh]

OK, thanks again to everyone for the valuable feedback. I now settled for "not yet another product" and set up and configured full text indexing for my Nextcloud and installed a better PDF and ePub reader app on Nextcloud. That combined with a folder structure that now need not be perfect - because full text index - is probably going to do it.

The full text index based on Elastic even catches the ePubs from my Calibre library

I have come to realise that for private data and projects it is probably best to use one product and see how far that takes you. With Nextcloud unfortunately it always takes a bit of work to find out if one particular "app" is really practical or just a proof of concept.

E.g.

- Wiki ("Collectives") - limited but works for me. I ran Confluence on prem before.
- Todo/Kanban ("Deck") - no way, even for personal use e.g. Vikunja is far superior.
- PDF-Documents - well, for invoices, contracts, etc. I run Paperless-NGX, anyway, but that is also not well suited to papers and presentations just like (for me!) Calibre. So Nextcloud it is.
- etc.

@meeera When I first looked at Devonthink my immediate reaction was "yeah, sort of like Paperless-NGX". It's so easy to spin up yet another piece of software, and there is so much duplicated functionality, it's really overwhelming some times.

Kind regards,
Patrick