Large library in KOReader - Page 2

ctop · 02-03-2020, 11:16 PM

Quote:

Originally Posted by NiLuJe

A file search is going to be atrociously slow no matter what you do. On the upside, it is recursive.

Actually compared to the display in the file manager, the search is blazingly fast in my case. However, on the Kobo I have no reorganized the Library slightly and do see great speed improvements for the file manager display.

Now, while it is possible to use the file search instead of the file manager, one thing that can not be achieved here is adding a book to favorites. For me, that would speed up things even more, since I could mark books as favorites and access them quickly. However, as far as I can see, the only way to add to favorites is a long press on the entry in the file manager. I wish there was a menu entry or assignable command "Add to favorites" somewhere...
All the best,
Ctop

NiLuJe · 02-04-2020, 01:07 AM

Hmm. Define "blazingly fast" ^^.

And, just to double-check, we *are* talking about the "Find a file" menu entry, right?

FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).

ctop · 02-04-2020, 06:45 AM

Quote:

Originally Posted by NiLuJe

Hmm. Define "blazingly fast" ^^.

And, just to double-check, we *are* talking about the "Find a file" menu entry, right?

FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).

Well, as I said I have to wait more than 10 minutes for the file manager to come up, so 10s *is* fast. But it is good to know that Calibre metadata based search is even faster, I will try to use that then.

All the best,
Ctop

pazos · 02-04-2020, 09:47 AM

Quote:

Originally Posted by ctop

Well, as I said I have to wait more than 10 minutes for the file manager to come up, so 10s *is* fast. But it is good to know that Calibre metadata based search is even faster, I will try to use that then.

All the best,
Ctop

10 seconds is slow, 10 minutes is insane.

I'm aware that likebooks are really broken in many aspects:
- Its launcher have a bunch of memory leaks, which is specially bad on a home launcher. - Its media scanner is pretty broken and won't pause during mtp transfer.
- Its media scanner works per folder, not per document. If it crash it will need to start again in the same folder. It crashes a lot.

But the forma should be pretty responsive, in spite of vfat.

My suggestion would be to focus on the forma: remove the koreader installation (on .adds/koreader, keep a backup somewhere else if you like) and start from scratch. Do not attempt to index anything and report back timmings.

Frenzie · 02-04-2020, 12:55 PM

Quote:

Originally Posted by NiLuJe

FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).

That sounds excruciatingly slow. Atm I have "602 items, totaling 1.2 GiB" in my biggest folder and that takes less than a second to search through. It's not a flat file structure, but I would think flat should be faster if anything.

NiLuJe · 02-04-2020, 02:03 PM

@Frenzie: I wouldn't be surprised if a flat structure was faster, actually

.

I also have a bunch of extremely nasty trees (Hi, Python!) in various hidden dot folders, which we might be traversing? That might explain the discrepancy.

EDIT: Yup, probably. It should probably honor our "show hidden folders" setting to prevent that.

@pazos/@ctop: Yep, the Forma has *much* better storage.

Frenzie · 02-04-2020, 03:22 PM

Your "actually" says exactly what I said. :P Unless you meant to say you wouldn't be surprised if flat were slower.

I shouldn't have more than about 50-100 documents per folder max. But anyway, still a total of 600 and not quite instantaneous but close enough (a few 100 ms).

NiLuJe · 02-04-2020, 04:02 PM

@Frenzie: Oops, read that as "wouldn't"

.

NiLuJe · 02-04-2020, 04:29 PM

Yep, dot folders indeed: much faster on my end after https://github.com/koreader/koreader/pull/5816

.

ctop · 02-05-2020, 01:16 AM

OK, so I am making some progress here (still working on the Likebook for the moment). I now rearranged the library on the device in the way suggested here and I now see the file manager screen coming up almost immediately, which is great!
Now I am experimenting with the calibre search, which I previously could not use, since it is only available on the file manager search menu, which took more than 10 minutes to appear.

I did put part of my library on the SD-card, another part in the main memory. When I connect calibre, I can see all of them. On the first connect, it took about an hour to read the device, but now I am down to 2 minutes on every connect, so I am assuming Calibre uses a catalog on the device to speed things up. Calibre recognizes the main memory and the card separately and displays the list of books in there also separately.

From the experiments I made so far is that KOReader's calibre search is using only the catalog on the main device, since I can not find books that are on the SD-card. Is that correct? Is there a way to instruct KOReader to use both catalogs? Or to switch between them? Or do I have to put everything on the main memory in order to be found?

I just want to make sure I understand how things work before committing to one way or another.

Thanks to all the help so far,

Ctop

Frenzie · 02-05-2020, 05:36 AM

It looks like you can set two paths in your defaults.persistent.lua:

https://github.com/koreader/koreader....lua#L220-L226

Code:

-- Set a path to a folder that is filled by Calibre (must contain the file metadata.calibre)
-- e.g.
-- "/mnt/sd/.hidden" for Kobo with files in ".hidden" on the SD card
-- "/mnt/onboard/MyPath" for Kobo with files in "MyPath" on the device itself
-- "/mnt/us/documents/" for Kindle files in folder "documents"
SEARCH_LIBRARY_PATH  = ""
SEARCH_LIBRARY_PATH2 = ""

NiLuJe · 02-05-2020, 06:39 AM

Yep, you'll have to set *both* paths manually, as @Frenzie suggested.

The autodetection only picks up the main memory otherwise, as you've found out.

Frenzie · 02-05-2020, 11:09 AM

Quote:

Originally Posted by ilovejedd

It seems when you open a "parent" folder, the "child" folders also get scanned so it can display how many items/"grandchildren" there are in the the "child" folders. That said, even with 3125 "grandchildren" inside 27 "child" folders, opening the parent folder didn't take too long (2-3 seconds).

I thought you were (slightly) wrong, but upon further testing I realized you weren't. The folder above my 10k file testing folder was just as slow as the folder itself. To fix that I propose this change (or something with the same end result) to make it work how I had assumed it worked: https://github.com/koreader/koreader/pull/5819

The difference lies between just providing a rough count and actually "scanning" the attributes, whether the file's been opened, etc. of the files in the subfolder. Just giving a count is orders of magnitude faster.

Actual display could also be made rather significantly faster, at least for sorting methods like filename and date modified, but that'd require a more involved rewrite to grab such info more dynamically per page instead of all at once.

Imo there's a very good reason that no one ever noticed this performance issue though, because like I said over a 1000 files in one folder seems fairly annoying to manage even when it's fast. (Or at least that's the case for the 10k files that cause a noticeable half-second delay on my laptop.)

ctop · 02-08-2020, 06:44 AM

Quote:

Originally Posted by Frenzie

It looks like you can set two paths in your defaults.persistent.lua:

https://github.com/koreader/koreader....lua#L220-L226

Code:

-- Set a path to a folder that is filled by Calibre (must contain the file metadata.calibre)
-- e.g.
-- "/mnt/sd/.hidden" for Kobo with files in ".hidden" on the SD card
-- "/mnt/onboard/MyPath" for Kobo with files in "MyPath" on the device itself
-- "/mnt/us/documents/" for Kindle files in folder "documents"
SEARCH_LIBRARY_PATH  = ""
SEARCH_LIBRARY_PATH2 = ""

OK, it took me a good while to set this up, since Calibre kept crashing on me, but now I have everything in place and can confirm that it works just great! And this search is really fast as well. So I am very happy with how things worked out in the end.

All the best and thanks to everyone!

Ctop

Frenzie · 02-08-2020, 08:43 AM

As an aside, I've since tracked down a few choice performance optimizations:

Just giving a count should be very fast now: https://github.com/koreader/koreader/pull/5819 That is, subfolders won't significantly slow down the folder they're in anymore.

The most taxing part of the code for displaying folders is ideally ~20% faster: https://github.com/koreader/koreader/pull/5827 The percentage is based on my test with 10k files. Also the performance improvement is less for the whole, that's just for that specific loop. But that loop is one of the slowest parts.

02-04-2020, 02:03 PM	#21
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@Frenzie: I wouldn't be surprised if a flat structure was faster, actually . I also have a bunch of extremely nasty trees (Hi, Python!) in various hidden dot folders, which we might be traversing? That might explain the discrepancy. EDIT: Yup, probably. It should probably honor our "show hidden folders" setting to prevent that. @pazos/@ctop: Yep, the Forma has much better storage. Last edited by NiLuJe; 02-04-2020 at 02:07 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How large is your Calibre library?	calibrejunkie	Calibre	78	07-30-2021 07:25 PM
KOReader doesn't open large, complex ebooks?	SbAgOoTs	KOReader	33	01-13-2020 12:40 PM
KoReader folder bug see on Nickel library	anki	Kobo Reader	4	12-28-2019 07:24 PM
Creating a dedicated library partition for koreader.	Markismus	Kobo Developer's Corner	6	09-02-2013 09:02 AM
Best way to get a large Calibre library into the PE library?	Filark	enTourage Archive	0	04-20-2011 10:18 PM

02-04-2020, 01:07 AM	#17
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Hmm. Define "blazingly fast" ^^. And, just to double-check, we are talking about the "Find a file" menu entry, right? FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).

02-04-2020, 03:22 PM	#22
Frenzie Wizard Posts: 1,618 Karma: 724945 Join Date: Oct 2014 Location: Antwerp Device: Kobo Aura H2O	Your "actually" says exactly what I said. :P Unless you meant to say you wouldn't be surprised if flat were slower. I shouldn't have more than about 50-100 documents per folder max. But anyway, still a total of 600 and not quite instantaneous but close enough (a few 100 ms).

02-04-2020, 04:02 PM	#23
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@Frenzie: Oops, read that as "wouldn't" .

02-04-2020, 04:29 PM	#24
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Yep, dot folders indeed: much faster on my end after https://github.com/koreader/koreader/pull/5816 .

02-05-2020, 01:16 AM	#25
ctop Connoisseur Posts: 57 Karma: 43710 Join Date: Jun 2008 Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares	OK, so I am making some progress here (still working on the Likebook for the moment). I now rearranged the library on the device in the way suggested here and I now see the file manager screen coming up almost immediately, which is great! Now I am experimenting with the calibre search, which I previously could not use, since it is only available on the file manager search menu, which took more than 10 minutes to appear. I did put part of my library on the SD-card, another part in the main memory. When I connect calibre, I can see all of them. On the first connect, it took about an hour to read the device, but now I am down to 2 minutes on every connect, so I am assuming Calibre uses a catalog on the device to speed things up. Calibre recognizes the main memory and the card separately and displays the list of books in there also separately. From the experiments I made so far is that KOReader's calibre search is using only the catalog on the main device, since I can not find books that are on the SD-card. Is that correct? Is there a way to instruct KOReader to use both catalogs? Or to switch between them? Or do I have to put everything on the main memory in order to be found? I just want to make sure I understand how things work before committing to one way or another. Thanks to all the help so far, Ctop

02-05-2020, 06:39 AM	#27
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Yep, you'll have to set both paths manually, as @Frenzie suggested. The autodetection only picks up the main memory otherwise, as you've found out.

02-08-2020, 08:43 AM	#30
Frenzie Wizard Posts: 1,618 Karma: 724945 Join Date: Oct 2014 Location: Antwerp Device: Kobo Aura H2O	As an aside, I've since tracked down a few choice performance optimizations: Just giving a count should be very fast now: https://github.com/koreader/koreader/pull/5819 That is, subfolders won't significantly slow down the folder they're in anymore. The most taxing part of the code for displaying folders is ideally ~20% faster: https://github.com/koreader/koreader/pull/5827 The percentage is based on my test with 10k files. Also the performance improvement is less for the whole, that's just for that specific loop. But that loop is one of the slowest parts.