Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > KOReader

Notices

Reply
 
Thread Tools Search this Thread
Old 02-04-2020, 12:16 AM   #16
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
Quote:
Originally Posted by NiLuJe View Post
A file search is going to be atrociously slow no matter what you do. On the upside, it is recursive.
Actually compared to the display in the file manager, the search is blazingly fast in my case. However, on the Kobo I have no reorganized the Library slightly and do see great speed improvements for the file manager display.

Now, while it is possible to use the file search instead of the file manager, one thing that can not be achieved here is adding a book to favorites. For me, that would speed up things even more, since I could mark books as favorites and access them quickly. However, as far as I can see, the only way to add to favorites is a long press on the entry in the file manager. I wish there was a menu entry or assignable command "Add to favorites" somewhere...
All the best,
Ctop
ctop is offline   Reply With Quote
Old 02-04-2020, 02:07 AM   #17
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
Hmm. Define "blazingly fast" ^^.

And, just to double-check, we *are* talking about the "Find a file" menu entry, right?

FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).
NiLuJe is online now   Reply With Quote
Advert
Old 02-04-2020, 07:45 AM   #18
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
Quote:
Originally Posted by NiLuJe View Post
Hmm. Define "blazingly fast" ^^.

And, just to double-check, we *are* talking about the "Find a file" menu entry, right?

FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).
Well, as I said I have to wait more than 10 minutes for the file manager to come up, so 10s *is* fast. But it is good to know that Calibre metadata based search is even faster, I will try to use that then.

All the best,
Ctop
ctop is offline   Reply With Quote
Old 02-04-2020, 10:47 AM   #19
pazos
cosiñeiro
pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.
 
Posts: 369
Karma: 387867
Join Date: Apr 2014
Device: BQ Cervantes 4
Quote:
Originally Posted by ctop View Post
Well, as I said I have to wait more than 10 minutes for the file manager to come up, so 10s *is* fast. But it is good to know that Calibre metadata based search is even faster, I will try to use that then.

All the best,
Ctop
10 seconds is slow, 10 minutes is insane.

I'm aware that likebooks are really broken in many aspects:
- Its launcher have a bunch of memory leaks, which is specially bad on a home launcher. - Its media scanner is pretty broken and won't pause during mtp transfer.
- Its media scanner works per folder, not per document. If it crash it will need to start again in the same folder. It crashes a lot.

But the forma should be pretty responsive, in spite of vfat.

My suggestion would be to focus on the forma: remove the koreader installation (on .adds/koreader, keep a backup somewhere else if you like) and start from scratch. Do not attempt to index anything and report back timmings.
pazos is offline   Reply With Quote
Old 02-04-2020, 01:55 PM   #20
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
Quote:
Originally Posted by NiLuJe View Post
FWIW, on an H2O (whose storage speed sucks ass) w/ ~500 boks (using a classic LN, FN/ single-level hierarchy, i.e., 35 pages), a file search (starting from the Home dir) takes between 6 and 10s, and a Calibre search is instantaneous (and more accurate ;p).
That sounds excruciatingly slow. Atm I have "602 items, totaling 1.2 GiB" in my biggest folder and that takes less than a second to search through. It's not a flat file structure, but I would think flat should be faster if anything.

Last edited by Frenzie; 02-04-2020 at 01:57 PM.
Frenzie is offline   Reply With Quote
Advert
Old 02-04-2020, 03:03 PM   #21
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
@Frenzie: I wouldn't be surprised if a flat structure was faster, actually .

I also have a bunch of extremely nasty trees (Hi, Python!) in various hidden dot folders, which we might be traversing? That might explain the discrepancy.

EDIT: Yup, probably. It should probably honor our "show hidden folders" setting to prevent that.

@pazos/@ctop: Yep, the Forma has *much* better storage.

Last edited by NiLuJe; 02-04-2020 at 03:07 PM.
NiLuJe is online now   Reply With Quote
Old 02-04-2020, 04:22 PM   #22
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
Your "actually" says exactly what I said. :P Unless you meant to say you wouldn't be surprised if flat were slower.

I shouldn't have more than about 50-100 documents per folder max. But anyway, still a total of 600 and not quite instantaneous but close enough (a few 100 ms).
Frenzie is offline   Reply With Quote
Old 02-04-2020, 05:02 PM   #23
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
@Frenzie: Oops, read that as "wouldn't" .
NiLuJe is online now   Reply With Quote
Old 02-04-2020, 05:29 PM   #24
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
Yep, dot folders indeed: much faster on my end after https://github.com/koreader/koreader/pull/5816 .
NiLuJe is online now   Reply With Quote
Old 02-05-2020, 02:16 AM   #25
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
OK, so I am making some progress here (still working on the Likebook for the moment). I now rearranged the library on the device in the way suggested here and I now see the file manager screen coming up almost immediately, which is great!
Now I am experimenting with the calibre search, which I previously could not use, since it is only available on the file manager search menu, which took more than 10 minutes to appear.

I did put part of my library on the SD-card, another part in the main memory. When I connect calibre, I can see all of them. On the first connect, it took about an hour to read the device, but now I am down to 2 minutes on every connect, so I am assuming Calibre uses a catalog on the device to speed things up. Calibre recognizes the main memory and the card separately and displays the list of books in there also separately.

From the experiments I made so far is that KOReader's calibre search is using only the catalog on the main device, since I can not find books that are on the SD-card. Is that correct? Is there a way to instruct KOReader to use both catalogs? Or to switch between them? Or do I have to put everything on the main memory in order to be found?

I just want to make sure I understand how things work before committing to one way or another.

Thanks to all the help so far,

Ctop
ctop is offline   Reply With Quote
Old 02-05-2020, 06:36 AM   #26
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
It looks like you can set two paths in your defaults.persistent.lua:

https://github.com/koreader/koreader....lua#L220-L226
Code:
-- Set a path to a folder that is filled by Calibre (must contain the file metadata.calibre)
-- e.g.
-- "/mnt/sd/.hidden" for Kobo with files in ".hidden" on the SD card
-- "/mnt/onboard/MyPath" for Kobo with files in "MyPath" on the device itself
-- "/mnt/us/documents/" for Kindle files in folder "documents"
SEARCH_LIBRARY_PATH  = ""
SEARCH_LIBRARY_PATH2 = ""
Frenzie is offline   Reply With Quote
Old 02-05-2020, 07:39 AM   #27
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
Yep, you'll have to set *both* paths manually, as @Frenzie suggested.

The autodetection only picks up the main memory otherwise, as you've found out.
NiLuJe is online now   Reply With Quote
Old 02-05-2020, 12:09 PM   #28
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
Quote:
Originally Posted by ilovejedd View Post

It seems when you open a "parent" folder, the "child" folders also get scanned so it can display how many items/"grandchildren" there are in the the "child" folders. That said, even with 3125 "grandchildren" inside 27 "child" folders, opening the parent folder didn't take too long (2-3 seconds).
I thought you were (slightly) wrong, but upon further testing I realized you weren't. The folder above my 10k file testing folder was just as slow as the folder itself. To fix that I propose this change (or something with the same end result) to make it work how I had assumed it worked: https://github.com/koreader/koreader/pull/5819

The difference lies between just providing a rough count and actually "scanning" the attributes, whether the file's been opened, etc. of the files in the subfolder. Just giving a count is orders of magnitude faster.

Actual display could also be made rather significantly faster, at least for sorting methods like filename and date modified, but that'd require a more involved rewrite to grab such info more dynamically per page instead of all at once.

Imo there's a very good reason that no one ever noticed this performance issue though, because like I said over a 1000 files in one folder seems fairly annoying to manage even when it's fast. (Or at least that's the case for the 10k files that cause a noticeable half-second delay on my laptop.)
Frenzie is offline   Reply With Quote
Old 02-08-2020, 07:44 AM   #29
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
Quote:
Originally Posted by Frenzie View Post
It looks like you can set two paths in your defaults.persistent.lua:

https://github.com/koreader/koreader....lua#L220-L226
Code:
-- Set a path to a folder that is filled by Calibre (must contain the file metadata.calibre)
-- e.g.
-- "/mnt/sd/.hidden" for Kobo with files in ".hidden" on the SD card
-- "/mnt/onboard/MyPath" for Kobo with files in "MyPath" on the device itself
-- "/mnt/us/documents/" for Kindle files in folder "documents"
SEARCH_LIBRARY_PATH  = ""
SEARCH_LIBRARY_PATH2 = ""
OK, it took me a good while to set this up, since Calibre kept crashing on me, but now I have everything in place and can confirm that it works just great! And this search is really fast as well. So I am very happy with how things worked out in the end.

All the best and thanks to everyone!

Ctop
ctop is offline   Reply With Quote
Old 02-08-2020, 09:43 AM   #30
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
As an aside, I've since tracked down a few choice performance optimizations:

Just giving a count should be very fast now: https://github.com/koreader/koreader/pull/5819 That is, subfolders won't significantly slow down the folder they're in anymore.

The most taxing part of the code for displaying folders is ideally ~20% faster: https://github.com/koreader/koreader/pull/5827 The percentage is based on my test with 10k files. Also the performance improvement is less for the whole, that's just for that specific loop. But that loop is one of the slowest parts.
Frenzie is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
KOReader doesn't open large, complex ebooks? SbAgOoTs KOReader 33 01-13-2020 01:40 PM
KoReader folder bug see on Nickel library anki Kobo Reader 4 12-28-2019 08:24 PM
How large is your Calibre library? calibrejunkie Calibre 64 10-07-2019 11:25 PM
Creating a dedicated library partition for koreader. Markismus Kobo Developer's Corner 6 09-02-2013 10:02 AM
Best way to get a large Calibre library into the PE library? Filark enTourage Archive 0 04-20-2011 11:18 PM


All times are GMT -4. The time now is 02:16 AM.


MobileRead.com is a privately owned, operated and funded community.