Hash based metadata questions

nixR3ad · 01-19-2024, 09:01 PM

I am continually tweaking the storage layout for my books on the Kobo Sage I use to run koreader. Part of this is because, for organization, I use a top level {#shelf} which becomes the sorting directory for the titles. A book about pirates might be part of the #history shelf. After I have 5 or more pirate books, #pirates becomes its own shelf. I'm also routinely removing and restoring titles on the device, and moving around a bunch of sidecar files gets tiresome or I forget them. I also don't like to use any directories except for #shelves. This allows me to have my (trivial) cover/metadata view of the books without having to drill into author/title directories.

I decided to switch to hash-based metadata lookups. I have some questions.

1. I assume any writes to the file itself will result in a broken hash. Fixing epub structure, and writing covers into the documents... is this absolute and guaranteed to happen?

2. If I load an epub onto the kobo with calibre and open it, creating an entry in the metadata table, then I download the exact file, with no changes made from, the OPDS server, and open it that way, will there be a mismatch?

3. How can I move the metadata for a specific book to another device or implement it elsewhere? Does exporting the highlights work this way?

4. I understand writing highlights to a PDF will change its hash value. The guide mentions setting different metadata settings for these. I don't see where I can readily change the metadata storage default for a title or group of titles.

Has anyone used this method with much success or improvements to their setup?

hius07 · 01-19-2024, 11:20 PM

Changing a file content in any way will change its hash and metadata location.
You cannot have different metadata locations for groups. Once you open a book, its metadata is saved to the default location.

nixR3ad · 01-20-2024, 04:33 PM

Thank you. That is what I suspected. Is there any way to export the metadata AND re-associate it with the new file (after changes have been made)?

Also, and I'll take this one to the bug reports on GH, but I think indexing the new hash tables damaged the unit.

It might be a coincidence, but as soon as the option was selected and the scan initiated, the kobo crashed. I've rescued many soft-bricked readers over the last decade, but the device is unresponsive. When plugged into either a power brick or my PC, there is no indication of power or charge being taken.

Maybe it was too much for the CPU. I was using the nightly from 01-18-24 and had applied the BrowseByMetadata3 patches (though maybe not needed). I'm responsible, but this should be known. I'd only had the Sage for 28 days and got a refund, fortunately.

NiLuJe · 01-20-2024, 07:13 PM

Quote:

Originally Posted by nixR3ad

Also, and I'll take this one to the bug reports on GH, but I think indexing the new hash tables damaged the unit.

That's... extremely unlikely.

You haven't mentioned how many files we're talking about here, but, assuming "a lot", I imagine there could be a crash or a deadlock or an OOM (after all, this is an extremely niche feature that had to fight really really hard to even get in, as none of the maintainers will ever use it), but *killing* a device? Eeeeh.

Having recently done something dumb to a Sage to cause an OOM, I can attest that it gets very finicky, but it *will* ultimately hard poweroff and then power back on again properly.

If you've got the frontlight on, that's a pretty good indication of when a long-long-power-press took for the hard-poweroff.

At which point, let it sit for a short bit, and then power it back on again. That initial bootup may be wonkier or longer than usual (e.g., no boot animation).

(If it's in a PowerCover, take it out of it, that thing's a mess that horribly breaks PM in fun and interesting ways).

nixR3ad · 01-20-2024, 07:48 PM

I wasn't sure just how much processing power that index required. A lot of books is relatively low actually, 130 epubs. I feared maybe it got stuck in a loop or something and overheated.

I'll do the suggested steps. If I can get it to power up again, I'll bring my crash logs with the report.

Side note: I was trying to see where the BrowseByMetadata3 patch someone posted on GutHub was doing anything. I'm fairly new to using Koreader for epubs (prior it was only PDFs). So, there are still stock features I'm discovering all the time after about 3 weeks with it as my primary app.

I had no virtual folder views in the file manager anywhere I could detect. The file search feature had an "include metadata" option. Is that standard (with a recent nightly)? Either way, thanks for the software.

I'll report back on how the recovery goes. That Sage was luxurious, but battery life was terrible and I'm debating a Boox Note Air 2e next if it doesn't work, as much as I am leary of Boox's practices.

NiLuJe · 01-21-2024, 12:29 AM

Assuming you did an "Extract and cache book information", there's a progress report being displayed live for that

(If there's another specific thingy related to the hash stuff, ignore me; as I've mentioned, we're not familiar with it. But the actual hash is extremely simple, so, no, the main bottleneck is going to be I/O).

Also, before anyone else gets a bright idea: the metadata browsing patch is a patch because it's still a WIP. Its intended use-case is, at the time of writing, to gather feedback from people with an actually sane & fully-fledged setup. Ideally people vaguely familiar with how KOReader works

.
That also means that each iteration of it targets a *specific* version of KOReader. Do *not* take any liberties with that requirement.

nixR3ad · 01-22-2024, 10:03 AM

Yeah, the patch application was just me throwing darts in the dark and seeing what stuck. Boredom and curiosity more than a directed quest for specific functionality. That said, the fiasco brought me closer to understanding "how koreader works." 😀

The Sage came back up. It seems it was just a coincidental battery drain coupled with a (probable) issue I experienced with the Remarkable I briefly owned. The charger being used was a 45w fast charging block (US). I'd not read anything about the Sage, but when my Rm2 was at 0% battery, it went into a loop when on such a charger. That device has a known issue and won't accept a charge from a fast block when below a certain percentage. Just conjecture.

When I put the Sage on a slower charge, it booted up. Thanks again.

01-19-2024, 09:01 PM	#1
nixR3ad Enthusiast Posts: 25 Karma: 10 Join Date: Jan 2024 Device: Sage / rm2	Hash based metadata questions I am continually tweaking the storage layout for my books on the Kobo Sage I use to run koreader. Part of this is because, for organization, I use a top level {#shelf} which becomes the sorting directory for the titles. A book about pirates might be part of the #history shelf. After I have 5 or more pirate books, #pirates becomes its own shelf. I'm also routinely removing and restoring titles on the device, and moving around a bunch of sidecar files gets tiresome or I forget them. I also don't like to use any directories except for #shelves. This allows me to have my (trivial) cover/metadata view of the books without having to drill into author/title directories. I decided to switch to hash-based metadata lookups. I have some questions. 1. I assume any writes to the file itself will result in a broken hash. Fixing epub structure, and writing covers into the documents... is this absolute and guaranteed to happen? 2. If I load an epub onto the kobo with calibre and open it, creating an entry in the metadata table, then I download the exact file, with no changes made from, the OPDS server, and open it that way, will there be a mismatch? 3. How can I move the metadata for a specific book to another device or implement it elsewhere? Does exporting the highlights work this way? 4. I understand writing highlights to a PDF will change its hash value. The guide mentions setting different metadata settings for these. I don't see where I can readily change the metadata storage default for a title or group of titles. Has anyone used this method with much success or improvements to their setup?

01-21-2024, 12:29 AM	#6
NiLuJe BLAM! Posts: 13,478 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Assuming you did an "Extract and cache book information", there's a progress report being displayed live for that (If there's another specific thingy related to the hash stuff, ignore me; as I've mentioned, we're not familiar with it. But the actual hash is extremely simple, so, no, the main bottleneck is going to be I/O). Also, before anyone else gets a bright idea: the metadata browsing patch is a patch because it's still a WIP. Its intended use-case is, at the time of writing, to gather feedback from people with an actually sane & fully-fledged setup. Ideally people vaguely familiar with how KOReader works . That also means that each iteration of it targets a specific version of KOReader. Do not take any liberties with that requirement. Last edited by NiLuJe; 01-21-2024 at 12:40 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Get metadata from Libgen based on file md5sum	iamashwin99	Development	2	07-26-2020 12:46 PM
Send to device based on Metadata	DavidTC	Calibre	0	09-18-2011 01:18 PM
rename files based on metadata	jchrist	Calibre	10	09-14-2010 10:28 AM
UK based e-reader - questions from a newbie	Georgiegirl2012	General Discussions	18	09-10-2010 05:45 PM

01-19-2024, 11:20 PM	#2
hius07 Wizard Posts: 1,058 Karma: 3581267 Join Date: Sep 2014 Location: Ukraine Device: Kindle PW2, PW3, PW4, Voyage	Changing a file content in any way will change its hash and metadata location. You cannot have different metadata locations for groups. Once you open a book, its metadata is saved to the default location.

01-20-2024, 04:33 PM	#3
nixR3ad Enthusiast Posts: 25 Karma: 10 Join Date: Jan 2024 Device: Sage / rm2	Thank you. That is what I suspected. Is there any way to export the metadata AND re-associate it with the new file (after changes have been made)? Also, and I'll take this one to the bug reports on GH, but I think indexing the new hash tables damaged the unit. It might be a coincidence, but as soon as the option was selected and the scan initiated, the kobo crashed. I've rescued many soft-bricked readers over the last decade, but the device is unresponsive. When plugged into either a power brick or my PC, there is no indication of power or charge being taken. Maybe it was too much for the CPU. I was using the nightly from 01-18-24 and had applied the BrowseByMetadata3 patches (though maybe not needed). I'm responsible, but this should be known. I'd only had the Sage for 28 days and got a refund, fortunately.

01-20-2024, 07:48 PM	#5
nixR3ad Enthusiast Posts: 25 Karma: 10 Join Date: Jan 2024 Device: Sage / rm2	I wasn't sure just how much processing power that index required. A lot of books is relatively low actually, 130 epubs. I feared maybe it got stuck in a loop or something and overheated. I'll do the suggested steps. If I can get it to power up again, I'll bring my crash logs with the report. Side note: I was trying to see where the BrowseByMetadata3 patch someone posted on GutHub was doing anything. I'm fairly new to using Koreader for epubs (prior it was only PDFs). So, there are still stock features I'm discovering all the time after about 3 weeks with it as my primary app. I had no virtual folder views in the file manager anywhere I could detect. The file search feature had an "include metadata" option. Is that standard (with a recent nightly)? Either way, thanks for the software. I'll report back on how the recovery goes. That Sage was luxurious, but battery life was terrible and I'm debating a Boox Note Air 2e next if it doesn't work, as much as I am leary of Boox's practices.

01-22-2024, 10:03 AM	#7
nixR3ad Enthusiast Posts: 25 Karma: 10 Join Date: Jan 2024 Device: Sage / rm2	Yeah, the patch application was just me throwing darts in the dark and seeing what stuck. Boredom and curiosity more than a directed quest for specific functionality. That said, the fiasco brought me closer to understanding "how koreader works." 😀 The Sage came back up. It seems it was just a coincidental battery drain coupled with a (probable) issue I experienced with the Remarkable I briefly owned. The charger being used was a 45w fast charging block (US). I'd not read anything about the Sage, but when my Rm2 was at 0% battery, it went into a loop when on such a charger. That device has a known issue and won't accept a charge from a fast block when below a certain percentage. Just conjecture. When I put the Sage on a slower charge, it booted up. Thanks again.