Magic Memory Upgrade Mod

[frostschutz]

I didn't delete it... seems like I should choose another host next time.

[PA_kobo]

frostschutz, I think I might've posted this in the wrong place, so I'll quote the visitor message which I can't seem to delete (sorry, I made an account just to ask about this):

"I downloaded the entire English epub collection from Project Gutenberg, and was hoping to put it as well as a Wikipedia ZIM file using a pre-4.0 OS (as mentioned here https://www.mobileread.com/forums/sh...d.php?t=276219) on a 128GB external card in my H2O (I was able to find an older ZIM file of the ebook collection, but searching it like an offline Wikipedia isn't really practical as it doesn't remember its place in books or list them).

Using FAT32 the card can't handle that many files, so I discovered your hack to modify the filesystem, but then the sqlite file (I guess for indexing the books) on the internal card filled up the entire internal card. Presumably this is because there are too many files/books (I read someone had tried it with 10,000 or so, and that was already over 800 megabytes, while the Project Gutenberg collection is over 44,000 books). Without taking the H2O apart, it seems the only solution is to clone the internal card to the external one (perhaps using your magic memory hack) and somehow boot the stock OS from the external card expanding the data partition to fill the entire card.

Is this doable? Or is there another solution to the sqlite file becoming too large without taking the H2O apart and breaking the seal?"

[frostschutz]

FAT32 should be able to handle many files - maybe not all in one folder, not sure.

Still, that's a lot of books. Perhaps Koreader or Coolreader would handle it better. If you go by folder structure, there is no need for any database at all.

You should create a new thread for this problem. And perhaps check the SQLite database out to find out what is making it so huge. You can install SQLite related programs on PC as well.

[h87booky]

Originally Posted by PA_kobo

"I downloaded the entire English epub collection from Project Gutenberg, and was hoping to put it as well as a Wikipedia ZIM file using a pre-4.0 OS (as mentioned here https://www.mobileread.com/forums/sh...d.php?t=276219) on a 128GB external card in my H2O (I was able to find an older ZIM file of the ebook collection, but searching it like an offline Wikipedia isn't really practical as it doesn't remember its place in books or list them).

Hi,

Reading books inside a zim file is very tedious. Also it's a very very big file because the zim contains multiple ebooks formats (epub, html, pdf). However it's possible to extract the epubs inside.

gutenberg_en_all_10_2014.zim = 42.8 GB
epubs inside = 6.8 GB
You save 36 GB... for a total of 38731 ebooks!

To extract the content: http://www.openzim.org/wiki/ZimDump

After that you can sort the ebooks inside directories, for (dirty) example:

#!/bin/sh

./zimdump -D /your/working/directory gutenberg_en_all_10_2014.zim

cd /your/working/directory

for x in {A..Z}
do
    mkdir $x
done

for y in {A..Z}
do
    mv $y*.epub $y/
done

mkdir 0
for z in {0..9}
do
    mv $z*.epub 0/
done

mkdir _
mv *.epub _/

Then you read with KOReader... Or repack it with only the epubs! http://www.openzim.org/wiki/Build_your_ZIM_file

I know it's possible to download the Gutenberg Project with an epub filter, I never succeeded. So it was easier to download the already packed zim and to extract the epubs inside. And it's very painful to read books inside a zim file... And I use KOReader a lot...

I almost forgot... Kobo sqlite database could explode with all these ebooks, so I put them inside a hidden directory (Unix-like, which begins with a dot), ex: ".Gutenberg_EN/".

[PA_kobo]

frostschutz, sorry for posting on the wrong thread, then. Splitting the files into sub-directories did seem to allow them all to fit on the card, so thanks for that.
h87booky, I have succeeded in doing a wget (or a series of them) to download all the books with an epub (and English) filter. I can send you the tar if you want. But are you saying you made the directory hidden on the FAT32 partition? And do the books still get indexed into your kobo library that way? How do you find them to read them on the device if they're not in the database?
Edit: Or you can do the download yourself, but for me it took all day (adding a delay of 1.1 seconds between books). I used a variation on this script (the epubs aren't unzipped, for example, so I had to change that):
https://www.exratione.com/2014/11/ho...ect-gutenberg/

[Ken Maltby]

I, for one, find it difficult to fathom how you managed to have your H2O process 40,000+ ebooks, and survive. How long did it take? Is there any performance issue, with Nickel trying to manage your books? How do you find your books within Kobo's Library menu? How long does it take to resort the listing?

I hope you keep an uncorrupted copy of your "KoboReader.sqlite" handy.

It is pretty easy to find ebook files in a logically constructed file system, which is handy for browsing. If you wish to find books directly, KOReader has search functions. If you were to create a calibre library of your downloaded ebooks and had calibre send them to your device, there would be a metadata index file generated that KOReader can use to find books, using metadata tags.

Luck;
Ken

[h87booky]

The truth is that I don't use Nickel anymore (original Kobo ereader), except for specific cbr/cbz and for zim, offline wikipedia. I use KOReader. This is why I put most of my ebooks inside hidden directories, Nickel can't see them. The search with KOReader is surprisingly fast, even with thousands of books. The day KOReader will read zim files or will have a decent web browser to use kiwix-serve, I stop using Nickel.

Thank you for the Gutenberg download script.

Sorry to pollute the thread with off-topic.

Thank you for your Magic Memory Upgrade Mod, it will be useful some day.

[cramoisi]

just saying, years later, it's still the best thing to use for upgrade storage

[Derf]

Originally Posted by cramoisi

just saying, years later, it's still the best thing to use for upgrade storage

I'd like to use this... will the files provided work if the firmware has been updated to a newer version?

[Derf]

Uh Oh....


so i dowloaded the Kobo-MagicMemory-3.16.10.zip file, and extracted it, then rearchived (via CLI) it KoboRoot.tgz, then put it in the .kobo folder on my ClaraHD. The Clara has firmware 4.9.1131.

I ejected the Kobo and it automatically shut down to restart, the screen went white with a progress bar that loaded black dots, then white dots then black dots... for about 4 or 5 minutes. Then it just stopped updating the progress bar. The light on the power button stopped blinking. The screen never went completely Black. So... I'm at a loss for what to do. Its been half an hour.

So, I tried to shut it down. The clara doesn't have a factory reset switch, just a power button. I tried plugging the device into power and holding down the power button, as it describes here, but it never turned off.

I now can't get the white screen with black boxes off the screen.

[frostschutz]

> then rearchived (via CLI) it KoboRoot.tgz

What exactly do you mean by rearchive?

You don't have to archive anything, only copy and rename the tgz file.

Something went wrong here. The mod uninstalls itself (from disk) before doing anything else, so everything happening afterwards lives in RAM and should be gone after restarting. So a reboot should get it back to normal.

I haven't tested this mod recently and I don't have a Clara HD to test with in the first place.

[Derf]

I am on a Mac, and the file I downloaded was a zip. My unarchiver is a little over zealous and extracted the .tgz from the .zip and I was left with a folder called sbin.
You posted about this a few posts back. I opened a terminal and issued “tar -czf KoboRoot.tgz sbin” and created the tar file.

Things seemed to go according to how they should. It rebooted, and I got a loading progress bar for a few minutes. Then it just stopped.

Someone a few posts back also had this problem, but they discovered their issue was with the memory card not being fast enough. I already speed tested my card and it was fast in both 1M and 4K read/write speeds.

I’m not sure why it hung, but I just want to restart the device. Unfortunately it seems totally unresponsive to restarting. Holding the power button doesn’t turn it off. And there is no hole for a paperclip on the Clara.

[frostschutz]

If you upload the tar you made somewhere, I can check if it's correct.

Well, it would be better to use the provided tar file as is. Certainly there is an unzip for Mac that doesn't force you to unpack the tar as well?

I'm not familiar with the Clara, if you have opened the device anyway to get to the SD Card slot, you might be able to disconnect / reconnect the battery, if it's a plug like was the case with old Kobos, and not soldered/glued on.

It's also possible for the battery to drain, if it was stuck in a script somewhere. The Memory Mod does an infinite loop waiting for you to replace the SD card, but that's with sleep statements so it should not eat power. However there may still be Kobo services in the background and who knows what happens there, so maybe try charging it as well.

[NiLuJe]

Yeah macOS is terrible for this sort of things.

I'd wager you were missing a few bits from what the actual tarball should have looked like.

Or there's a new HW/SW quirk on these devices that we're not aware of yet that's upsetting this specific mod.

[Derf]

On a Mac when you double click a zip, there is a built in unarchiver. It isn’t like on windows where you have to download winzip and winrar.
I do have a background in Linux, and so instead of re finding the webpage with the link, re-downloading the zip, and using a command line to unzip the tar file, I just re-tared the folder that was extracted. Does that make sense?

Yes, I can upload a copy, but you’re right, it really is just better to use the file you have.

The battery IS with a connector and not soldered to the board. I could disconnect/reconnect it, but does it matter if I have my original card in the kobo when I try this? I don’t want it to try to format my card with the OS on it and try to copy a broken system onto it from RAM.