PLUGIN All eBooks in single flat view on KOreader folder structure independent

[PetBest]

This KOreader plugin organizes ALL your eBooks of type epub, pdf, awz3, mobi, docx into a single collection!
Many people asked for such solution on Reddit and other

It works independent on how you have organized the structure (all books in one folder or like the structure that Calibre by default creates on your reader.

It works as a normal KOreader Collection at is standard called 'All Books'
You can click on the 'hamburger - or three horizontal stripes at left upper corner' menu in the collection view to sort the collection list.

More than 10.000 ebooks in a few hundred folders of Authors is created very fast is a few seconds. After a restart the Collection is visible.

Under menu Tools (wrench/screwdriver icon) go to next page (page 2) and click on More Tools > Click on All eBooks Collection
This will start the function.
That function can be coupled to a gesture that you like. I used swipe-up movement on right edge of my reader screen and that starts the creation of the collection. Always restart KOreader after a new creation process to make the collection and its changes visible.
You can repeat this function without any problem.

Have FUN.

PS: I used inspiration from a similar solution that was created as a patch but that was always using Favorites and it had severe issues with large collections..only a few hundred books otherwise it crashed. Only epub and pdf were processed.
If you have other types like .txt that you want to make visible look into the source code and and that looks like this (find it around lines 160-165):

local pfile = popen('find "'..directory..'" -maxdepth 10 -type f -name "*.epub" -o -name "*.pdf" -o -name "*.azw3" -o -name "*.mobi" -o -name "*.DOCX" | sort ')

So if you want .txt then it will look like this

local pfile = popen('find "'..directory..'" -maxdepth 10 -type f -name "*.epub" -o -name "*.pdf" -o -name "*.azw3" -o -name "*.mobi" -o -name "*.DOCX" -o -name "*.txt" | sort ')

Note: I have tested it on Ubuntu and Android. i expect it will also work on Kobo as I use standard functions of KOreader.

DOWNLOAD AT
https://github.com/peterstamps/All_e...ew_on_KOreader

[Billiam]

Looks neat! For larger sets of books like that, how do you go about finding your books in collections without a search or filter?

[PetBest]

There are various ways.
1. Just scrolling through lists with mosaic display on till cover is recognised
2. Searching using Calibre metadata search after indexing device when starting this function( used in combination with Calibre)
3 File search
4 I installed a Patch called BrowseByMetadata found here https://github.com/user-attachments/...-userpatch.zip

[PetBest]

Just create a folder called patches in the Koreader (At the samedirecty level as plugins folder) and put there the lua file that you find in the zip file

[Billiam]

Sure, but 2, 3, and 4 require you to exit the collection view, no?

[PetBest]

Correct.

[gereksizuser]

This is great, I modified the main.lua file a bit with the help of AI and now I can group by authors. My next stop will be to create collections by Genre. Thanks a lot.

[PetBest]

Can you share your modification?
Great idea!

I tried/, try to figure out how to update immediately the collection without a restart.
So far without success. If you find a way to do it please share.

[PetBest]

On which device are you running KOreader.
Is that a Kobo Libra 2?

[gereksizuser]

Originally Posted by PetBest

Can you share your modification?
Great idea!

I tried/, try to figure out how to update immediately the collection without a restart.
So far without success. If you find a way to do it please share.

Originally Posted by PetBest

On which device are you running KOreader.
Is that a Kobo Libra 2?

I'm using the Kobo Aura n236, here it is;

-- KOReader plugin to add all ebooks found on device to separate collections by author
require "defaults"
package.path = "common/?.lua;frontend/?.lua;" .. package.path
package.cpath = "common/?.so;common/?.dll;/usr/lib/lua/?.so;" .. package.cpath

local Dispatcher = require("dispatcher")  -- luacheck:ignore
local InfoMessage = require("ui/widget/infomessage")
local UIManager = require("ui/uimanager")
local WidgetContainer = require("ui/widget/container/widgetcontainer")
local _ = require("gettext")
local LuaSettings = require("luasettings")
local ffiUtil = require("ffi/util")
local lfs = require("libs/libkoreader-lfs")
local logger = require("logger")
local util = require("util")
local DataStorage = require("datastorage")
local ReadCollection = require("readcollection")
local FileManagerCollection = require("apps/filemanager/filemanagercollection")
local DocumentRegistry = require("document/documentregistry")

if G_reader_settings == nil then
    G_reader_settings = require("luasettings"):open(
        DataStorage:getDataDir().."/settings.reader.lua")
end

-- Get the base ebooks folder on the ereader to start the search for ebooks
local ebooks_directory_path = G_reader_settings:readSetting("home_dir")

-- Get the Koreader base folder on the ereader where the settings are stored
local koreader_settings_directory_path = DataStorage:getFullDataDir()

-- Get the Koreader file on the ereader where the settings are stored for collections
local collection_file = DataStorage:getSettingsDir() .. "/collection.lua"

local ReadCollection = {
    coll = nil, -- hash table
    coll_settings = nil, -- hash table
    last_read_time = 0,
    default_collection_name = "favorites",
}

-- read, write

local function buildEntry(file, order, attr)
    file = ffiUtil.realpath(file)
    if file then
        attr = attr or lfs.attributes(file)
        if attr and attr.mode == "file" then
            return {
                file  = file,
                text  = file:gsub(".*/", ""),
                order = order,
                attr  = attr,
            }
        end
    end
end

function ReadCollection:_read()
    local collection_file_modification_time = lfs.attributes(collection_file, "modification")
    if collection_file_modification_time then
        if collection_file_modification_time <= self.last_read_time then return end
        self.last_read_time = collection_file_modification_time
    end
    local collections = LuaSettings:open(collection_file)
    if collections:hasNot(self.default_collection_name) then
        collections:saveSetting(self.default_collection_name, {})
    end
    logger.dbg("ReadCollection: reading from collection file")
    self.coll = {}
    self.coll_settings = {}
    for coll_name, collection in pairs(collections.data) do
        local coll = {}
        for _, v in ipairs(collection) do
            local item = buildEntry(v.file, v.order)
            if item then -- exclude deleted files
                coll[item.file] = item
            end
        end
        self.coll[coll_name] = coll
        self.coll_settings[coll_name] = collection.settings or { order = 1 } -- favorites, first run
    end
end

function ReadCollection:write(updated_collections)
    local collections = LuaSettings:open(collection_file)
    for coll_name in pairs(collections.data) do
        if not self.coll[coll_name] then
           collections:delSetting(coll_name)
        end
    end
    for coll_name, coll in pairs(self.coll) do
        if updated_collections == nil or updated_collections[1] or updated_collections[coll_name] then
            local is_manual_collate = not self.coll_settings[coll_name].collate or nil
            local data = { settings = self.coll_settings[coll_name] }
            for _, item in pairs(coll) do
                table.insert(data, { file = item.file, order = is_manual_collate and item.order })
            end
            collections:saveSetting(coll_name, data)
        end
    end
    logger.dbg("ReadCollection: writing to collection file")
    collections:flush()
end

-- info

function ReadCollection:isFileInCollection(file, collection_name)
    file = ffiUtil.realpath(file) or file
    return self.coll[collection_name][file] and true or false
end

function ReadCollection:getCollectionNextOrder(collection_name)
    if self.coll_settings[collection_name].collate then return end
    local max_order = 0
    for _, item in pairs(self.coll[collection_name]) do
        if max_order < item.order then
            max_order = item.order
        end
    end
    return max_order + 1
end

-- manage items

function ReadCollection:addItem(file, collection_name)
    local item = buildEntry(file, self:getCollectionNextOrder(collection_name))
    self.coll[collection_name][item.file] = item
end

-- manage collections

function ReadCollection:addCollection(coll_name)
    local max_order = 0
    for _, settings in pairs(self.coll_settings) do
        if max_order < settings.order then
            max_order = settings.order
        end
    end
    self.coll_settings[coll_name] = { order = max_order + 1 }
    self.coll[coll_name] = {}
end

-- Extract author name from metadata or file name
local function extractAuthor(file_path)
    -- Try to get author from metadata
    local doc = DocumentRegistry:openDocument(file_path)
    if doc then
        local metadata = doc:getProps()
        if metadata and metadata.author then
            local meta_author = metadata.author
            doc:close()
            return meta_author
        end
        doc:close()
    end

    -- Fallback to extracting author from file name
    local file_name = file_path:match(".*/([^/]+)$") -- Extract file name from path
    if file_name then
        local author, _ = file_name:match("^(.-) %- (.+)$") -- Split by " - "
        if author then
            return author:gsub("%s+$", "") -- Trim trailing spaces
        end
    end
    return "Unknown Author" -- Default if format is not matched
end

local AllMyeBooks = WidgetContainer:extend{
    name = "All eBooks",
    is_doc_only = false,
}

function start(ebooks_directory_path)
    local files = scandir(ebooks_directory_path)
    ReadCollection:_read()
    
    -- Group files by author
    local books_by_author = {}
    for _, file in ipairs(files) do
        local author = extractAuthor(file)
        if not books_by_author[author] then
            books_by_author[author] = {}
        end
        table.insert(books_by_author[author], file)
    end
    
    -- Sort authors alphabetically
    local sorted_authors = {}
    for author in pairs(books_by_author) do
        table.insert(sorted_authors, author)
    end
    table.sort(sorted_authors, function(a, b) return a < b end)
    
    -- Create a collection for each author and add their books
    for _, author in ipairs(sorted_authors) do
        local collection_name = author
        ReadCollection:addCollection(collection_name)
        add_ebooks_to_collection(books_by_author[author], collection_name)
    end
    
    UIManager:show(InfoMessage:new{
        text = _("Collections by author have been created. Please restart KOReader for changes to take effect."),
        timeout=9,
    })
end

function scandir(directory)
    local i, t, popen = 0, {}, io.popen
    local pfile = popen('find "'..directory..'" -maxdepth 10 -type f  -name "*.epub" -o -name "*.pdf" -o -name "*.azw3" -o -name "*.mobi" -o -name "*.docx" -o -name "*.cbz"  | sort ') -- on linux
    for filename in pfile:lines() do
        i = i + 1
        t[i] = filename   
    end
    pfile:close()
    return t
end

function add_ebooks_to_collection(files, collection_name) 
    for _, file in ipairs(files) do
        ReadCollection:addItem(file, collection_name)   
    end
    ReadCollection:write({collection_name})
end

function AllMyeBooks:onDispatcherRegisterActions()
    Dispatcher:registerAction("AllMyeBooks_action", {category="none", event="AllMyeBooks", title=_("Create Collections by Author"), general=true,})
end

function AllMyeBooks:init()
    self:onDispatcherRegisterActions()
    self.ui.menu:registerToMainMenu(self)
end

function AllMyeBooks:addToMainMenu(menu_items)
    menu_items.AllMyeBooks = {
        text = _("Create Collections by Author"),
        sorting_hint = "more_tools",
        callback = function()
            UIManager:show(InfoMessage:new{
                text = _("Creating collections by author... Please wait."),
                timeout=5,
            })
            start(ebooks_directory_path)
        end,
    }
end

function AllMyeBooks:onAllMyeBooks()
    local popup = InfoMessage:new{
      text = _("Creating collections by author... Please wait."),
      timeout=5,
    }
    UIManager:show(popup)
    start(ebooks_directory_path)
    local popup = InfoMessage:new{
      text = _("Collections by author have been created. Please restart KOReader for changes to take effect."),
      timeout=9,
    }
    UIManager:show(popup)
end

return AllMyeBooks

Takes a little longer, but works great on my device.

[PetBest]

Thank you for sharing.
I tested it on my Android.
I got for each book a collection. Reason the file structure was:
title - author (calibre id).epub
Like: Raven - A.C.Writer (678).epub

But I had also a subfolder with author - title.epub.
The books came from various sources.
So it had to strip author like:

if author then
return author:gsub("%s+%((.-)$", "") -- Trim trailing spaces


It should be possible to show a Busy...please wait popup at the start, so you see that the process is ongoing.

BTW: some plus more of these features will be build in the core
see here: https://github.com/koreader/koreader...ent-2690334308

[PetBest]

Forgot to mention that the result was somewhat awkward as the base input was mixed w.r.t. structure.
The idea of a Collection per author is fine.
I saw that these collections were gone after a restart and I had to recreate them again.

[gereksizuser]

Sorry, I forgot to mention that my books are in the order of "Author name (FN>LN) - Book name", so I had them sorted in this order. The - sign is also mandatory to find the author's name.

[PetBest]

New KOReader plugin: Collection All my eBooks and/or By Author


Download at https://github.com/peterstamps/Flat-...-and-By-Author

This NEW KOReader plugin organizes ALL your eBooks of type epub, pdf, awz3, mobi, docx into Collection(s).
Many people asked for such solution on Reddit and other forums.

The following functions/choices are possible and Optional.

For All functions (1,2 and 3): First set your Home folder under the + Menu or by long tapping on a Folder! Otherwise the runtime folder (.) will be used!!

1. One single collection (flat view) with All eBooks.
See below Note B.
More than 10.000 ebooks are created very fast in just a few seconds. After a restart of KOReader the Collection 'All eBooks' becomes visible.

The choices 2 and 3 may result in many collections and may take quite some time if you have large collections (rule of thumb >500 ebooks)

2. Create by Author a Collection based on meta data.
All available books of the same Author(s) on your ereader will be stored in a single collection.
The Author name will be based on Metadata details taken from the document when available else the Author name is taken from the file name (see below Note A).

3. Create by Author a Collection based on file name.
All available books of the same Author(s) on your ereader will be stored in a single collection.
The Author name will only be taken from the file name (see below Note A). If you have used Calibre to save all your ebooks on the ereader and you have standardized/cleaned the Author names with Calibre then THIS choice is preferred above choice 2.

4. Remove All Collections except Favorites
Be carefull to use this function. ONLY Favorites will not be deleted and the content will stay as is. If you have manually created your own Collections and you want to keep them dan NEVER use this function as they will be removed!
If you have only Favorites or some small easy to recreate Collections, then you can use this function and recreate the Collections with the choices 1, 2, 3 shown above.

When using choices 2 AND 3 then one collection PER Author(s names) will be created, same for unique book. However if their are different spellings used for Authors (also between the filename and in the metadata) then ofcourse more collections for the same Author will appear.
So "A.L. Arlige" is NOT the same as "A.L.Arlige" and two collections will appear. That is also the case in Calibre and most eBook software.

NOTE A:
The default Calibre file name layout in the example below forms the basis for writing the program code for Author extraction:
EXAMPLE:
The default Calibre full filenames looks like this: "/Calibre Library Kobo - Test/John Doe/The Mysteries (21)/The Mysteries - John Doe.epub"
The Author will be taken from the last part of the full filename, here: "The Mysteries - John Doe.epub"
The part between the hyphen with space(s) and the file extention is considered to be the Author name, here: "John Doe". That will become the collection name!
If your eBook file names look like this "John Doe - The Mysteries.epub" or "John Doe - The Mysteries (21).epub" then you will get as Collection "The Mysteries"


NOTE B. ONLY for 'All eBooks' the Collection works independent on how you have organized the structure (all books in one folder so similar like the structure that Calibre by default creates on your reader).


All Collections work as normal KOReader Collections.
You can click on the 'hamburger - or three horizontal stripes at left upper corner' menu in the collection view to sort the collection list.


Under menu Library (cabinet icon) > Click on Create Collections
This will start the Submenu (Choice 1,2,3 and 4 see above).

Fuctions can be coupled to a gesture that you like.

I use swipe-up movement on right edge of my reader screen and that starts the creation of the 'All eBooks' collection.
I use swipe-down movement on right edge of my reader screen and that opens the list with collections.

Always restart KOreader after a new creation process to make the collection(s) and its changes visible.
You can repeat this function without any problem.



HOW TO USE GESTURES (> means click)
Menu Settings (wheel icon) > Taps and gestures > Gesture manager > One finger swipe > Right Edge Up > General > Page 2 (scroll) > CreateAllMyBooks Collection

HOW TO INSTALL
Copy the folder AllMyeBooks.koplugin under KOreaders plugin folder (search where KOreader is installed)
Unzip when needed if downloaded as zip file

The original PLUGIN with only the function 1 above is still available.

[PetBest]

I have corrected certain things w.r.t. metadata reading as that was not finding the authors in my epubs

I also combine the functions in one menu and now people have choices. Plus I added a remove function. That allows easy recreation when people use Calibre to add/delete books...