Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Related Tools

Notices

Reply
 
Thread Tools Search this Thread
Old 08-12-2024, 04:33 PM   #1
Pachuqismo
Enthusiast
Pachuqismo began at the beginning.
 
Posts: 37
Karma: 10
Join Date: Nov 2009
Device: SM-T820, Galaxy S7
Using GPT4All as a research tool

On my PC I've just installed GPT4All and selected as model the 'Llama 3 8B Instruct'. It is presently indexing my Calibre library to what it calls LocalDocs.

For anyone interested, I suggest to check it out at GPT4All's homepage (it's free and promises to fully respect privacy). It's not that much of an install and that way you can quiery the AI to your heart's content to get an idea of its use.

I expect it to be a significant tool to access specific information in my largely non-fiction library, but by no means do I consider it a replacement for Calibre for the simple reason that it is not - and doesn't pretend to be - a library manager.

Among the benefits listed by the LLM, I particularly paid attention to:

- Contextual search: allowing for additional questions and provision of additional context to refine results
- Natural Language Processing
- Integration with Chat interface

For anyone linking to their Calibre library, and who have many books, I suggest to create a sub-library in its own folder and run GPT4All on that. Do it with books that you know fairly well, so you have some possibility for evaluting produced results.

Checking GitHub and elsewhere, I understand that a GPT4All summarizer is in the pipeline, which would really be nice.

However, I would still carry on what I've done so far, which is to export my marked text to a text file (which my ePub reader allows to create and export). Then I use - say DuckDuckGo's Chat AI - to consolidate it into a coherent summary. From now on I expect to use GPT4All to do that.

I bet, at least for a long time, this sort of wetware curated summary (i.e. using my brain) will surpase what any AI have to offer, since who-better-to-know than I what is important for me.

Unless you're fanatically interested in certain topics and just prefer to enjoy fictions, all of this may surely not be important. If you're an (aspiring) writer then it may be different.

What I expect from GPT4All is to better understand the three existential threats presently facing humanity, which I find to be: Climate Change, Depletion of Biodiversity and Inequality at Local, Country and Global levels.

I got lots of relevant literature in my library but finding it is a challenge that I hope will be reduced significantly with the use of AI.

Cheers!
Pachuqismo is offline   Reply With Quote
Old 08-12-2024, 06:07 PM   #2
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 76,458
Karma: 136564696
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
I hope the answer is never to AI being added to calibre.
JSWolf is offline   Reply With Quote
Old 08-12-2024, 06:28 PM   #3
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,367
Karma: 92073397
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
Quote:
Originally Posted by Pachuqismo View Post
On my PC I've just installed GPT4All and selected as model the 'Llama 3 8B Instruct'. It is presently indexing my Calibre library to what it calls LocalDocs.

For anyone interested, I suggest to check it out at GPT4All's homepage (it's free and promises to fully respect privacy).
But the makers don't respect copyright. The whole subject is full of lies and anything useful is simply scraped. In reality it's a search engine that doesn't quote sources and plagiarises. It's also very inefficient and thus damaging to the environment.

Last edited by Quoth; 08-13-2024 at 10:28 AM.
Quoth is offline   Reply With Quote
Old 08-12-2024, 07:23 PM   #4
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,004
Karma: 27620706
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Moderator Notice

I moved this from the Calibre forum to the Related Tools subforum.

I have deleted the almost identical post in Plugins.

BR
BetterRed is online now   Reply With Quote
Old 08-13-2024, 10:31 AM   #5
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,367
Karma: 92073397
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
Calibre built-in Full Text Search (FTS) is very powerful, with phrase and near features.
You can run the search, when index is up to date, on a sub-range of titles by any normal Calibre search.

It's fantastic for research.
Quoth is offline   Reply With Quote
Old 08-15-2024, 04:18 PM   #6
Pachuqismo
Enthusiast
Pachuqismo began at the beginning.
 
Posts: 37
Karma: 10
Join Date: Nov 2009
Device: SM-T820, Galaxy S7
@Quoth

I'll be sure to check out the Calibre FTS results.

With respect to your previous response I think you're referring to the means used and present stranglehold Big Tech has on LLMs where they try to force people to use their cloud-based systems. I surely agree that this is a very negative development and that's why I'm keeping an eye on open source alternatives.

Also, AI has already had significant impact on various scientific domains enhancing research and practical applications. Just ask an AI
  1. Weather Prediction (1950s): Early AI algorithms improved meteorological forecasting. Reference: Charney & Shukla, 1981.
  2. Medical Imaging (1970s): AI began analyzing X-rays and CT scans to assist in disease diagnosis. Reference: Doi, 2007.
  3. Protein Folding (1990s): AI predicted protein structures, enhancing understanding of biological processes. Reference: Baker, 2000.
  4. Genomics (2000s): AI analyzed genomic data, advancing personalized medicine and genetic disease understanding. Reference: Shendure & Ji, 2008.
  5. Drug Discovery (2010s): AI models accelerated drug candidate identification and molecular interaction predictions. Reference: Vamathevan et al., 2019.
  6. Climate Modeling (2010s): AI improved climate change predictions and weather pattern analysis. Reference: Rasp et al., 2018.
  7. Radiology (2016): Deep learning models achieved radiologist-level accuracy in analyzing medical images. Reference: Esteva et al., 2017.
  8. COVID-19 Research (2020): AI analyzed COVID-19 data for outbreak predictions and vaccine development. Reference: Wang et al., 2020.
  9. AI in Astronomy (2020s): AI analyzed astronomical data, leading to new discoveries in celestial phenomena. Reference: McKinney et al., 2020.
  10. Environmental Science (2021): AI monitored environmental changes, aiding conservation efforts. Reference: Levin et al., 2021.
Pachuqismo is offline   Reply With Quote
Old 08-16-2024, 05:37 AM   #7
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,367
Karma: 92073397
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
Mostly lies and marketing. The so called AI of Watson to win Jeopardy was unrelated to the Watson IBM sold for Medicine, which didn't work.

Most of the so called success of AI is pattern-matching of human curated work branded as AI.

The LLM, on Cloud or local is even less use.

"Deep Learning" is a lie. AI Jargon along with terms like Machine Learning and Neural Networks (which aren't).

I started my study of programming due to interest in AI, which was marketed as Expert Systems from the late 1980s.

AI didn't do any of the things on your list. Pattern matching aided by computer databases and software from human curated sources achieved the results. Also none of that list applies to how LLM work.

AI after 60 years is still a toy, now being hyped like cryptocoins and NFT. Analysis suggests this is hype bubble and will fail as the costs of builing the LLM and feeding the data exceed any income possible, which is not even considering that the productivity gains claimed to sell it are by the people selling and are lies.


Misleading: https://www.theregister.com/2024/08/...y_creation_of/
This is like Big McD saying big Macs are healthy: https://www.theregister.com/2024/08/...ai_roi_report/
https://www.theregister.com/2024/08/..._going_to_pay/ This is separate to the question "Does it even work".

Last edited by Quoth; 08-16-2024 at 05:49 AM.
Quoth is offline   Reply With Quote
Old 08-23-2024, 01:18 PM   #8
Pachuqismo
Enthusiast
Pachuqismo began at the beginning.
 
Posts: 37
Karma: 10
Join Date: Nov 2009
Device: SM-T820, Galaxy S7
@Quoth

At least you're persistently negative in the subject matter

A shame that not a single of the 10 examples of useful application of AI are real and all part of a mayor conspiracy...
Pachuqismo is offline   Reply With Quote
Old 08-23-2024, 05:46 PM   #9
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,367
Karma: 92073397
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
The examples are real useful things, but not AI.
Quoth is offline   Reply With Quote
Old 08-31-2024, 05:03 PM   #10
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,367
Karma: 92073397
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
The LLM is a special case of "AI",- none of which is actual intelligence. Most is pattern matching.

This paper argues that the LLMs are over-hyped and the least useful "AI" because of poor reliability. They do not "hallucinate". That apparent behaviour is an inevitable failure.
Quote:
"There is no clear evidence that that shows LLMs are useful because they are extremely unreliable," Birhane said. "Various scholars have been doing domain specific audits … in legal space … and in medical space. The findings across all these domains is that LLMs are not actually that useful because they give you so much unreliable information."
https://www.theregister.com/2024/08/...arch/?td=rt-3a

And no-one knows if so called "general AI" is even possible, people that claim they do are lying or deluded or have no clue about intelligence.

As an aside, an IQ test is a comparative performance measurement on a narrow range of tasks that requires people to be at about the same age, educational, social and ethnic background. It doesn't measure intelligence and the originator never claimed it did, Mostly it's used to create social exclusion (NI UK Grammar schools, now abolished, or US Army).
Quoth is offline   Reply With Quote
Old 08-31-2024, 06:02 PM   #11
DNSB
Bibliophagist
DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.
 
DNSB's Avatar
 
Posts: 40,570
Karma: 157444380
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos
Quote:
Originally Posted by Pachuqismo View Post
Also, AI has already had significant impact on various scientific domains enhancing research and practical applications. Just ask an AI
  1. Weather Prediction (1950s): Early AI algorithms improved meteorological forecasting. Reference: Charney & Shukla, 1981.
I will admit to finding it humorous that someone would go back to the improvements brought by the use of computers to weather forecasting in the 1950s and attempt to use that as an example of artificial intelligence. Anyone else remember ENIAC and want to claim that it was an implementation of artificial intelligence? Take a look at M-ENIAC: A Physics-Informed Machine Learning Recreation of the First Successful Numerical Weather Forecasts. The implementation of the barotropic vorticity equation solver for cellphones was fun to play with considering the iPhone used for the implementation was ~10,000 times more power than the ENIAC.

Like most other fads in computer science, AI has become more an overused buzzword than anything meaningful.
DNSB is offline   Reply With Quote
Old 08-31-2024, 07:53 PM   #12
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,004
Karma: 27620706
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by DNSB View Post
. . .

Like most other fads in computer science, AI has become more an overused buzzword than anything meaningful.
We should have stuck to "Cybernetics". That was the term used when I attended lectures from Donald Michie at Edinburgh in the late 1960's. Amongst other things Michie was the developer of MENACE

BR
BetterRed is online now   Reply With Quote
Old 09-02-2024, 05:36 PM   #13
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,367
Karma: 92073397
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
Oh, I remember Cybernetics. It was a real thing and AI was only in SF.
Quoth is offline   Reply With Quote
Old 10-01-2024, 05:42 PM   #14
retval
Enthusiast
retval began at the beginning.
 
Posts: 34
Karma: 10
Join Date: Mar 2011
Device: none
You can detail the use cases and how the library is linked to the AI. Thanks

Quote:
Originally Posted by Pachuqismo View Post
For anyone who links to your Calibre library, and who has a lot of books, I suggest creating a sub-library in your own folder and running GPT4All on that. Do this with books you know fairly well, so you have some chance of evaluating the results produced. Cheers!
Thanks for your input, I find it very interesting that AI can be used with the Calibre library. I am also in research and for me calibre has been fundamental for years as a library organizer.
The full text search, integrated in Calibre, works very well. I have about 40,000 files in my library and Calibre processes the search quickly with only 16 gb of ram. It presents the results for the first 10,000 files containing the desired term in descending order.
Using Calibre together with Obsidian is very efficient.
Going back to your comment, it is not clear to me whether Chatgpt installs inside a new folder in the library or creates a folder with the books you want inside the ChatgptAll installation folder.
How do you, if you have 10 books, select one or two to chat with?
Contrary to what is stated, I think an add-on that integrates AI to Calibre would be fabulous for researchers. Calibre would be the perfect interface for working with large amounts of information.
Thanks again.
retval is offline   Reply With Quote
Old 10-05-2024, 01:43 AM   #15
Capricorn
Belgian Pommes Frites
Capricorn will become famous soon enoughCapricorn will become famous soon enoughCapricorn will become famous soon enoughCapricorn will become famous soon enoughCapricorn will become famous soon enoughCapricorn will become famous soon enough
 
Posts: 116
Karma: 532
Join Date: Jan 2012
Device: Pocketbook Touch HD
Weather forecasting: the physics model is basically based on 8 different algorithms that are fed phyical data like temperature,..etc. And produce the predictions that we know today (reliable for about three days). Hence, it is pure number crunching. And the more computing power, the better the predictions. This has nothing to do with AI.
AI in weather forecasting is based on previous weather patterns, and tries to predict the future, based on that. In other words, pattern matching. The problem is that climate disaster will change these patterns massively, especiall if the jet stream changes, hence the long-term validity of it, seems to be meager at the moment.
Capricorn is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Confused. Is Calibre an eBook creation Tool or tool for readers? Fran_3 Calibre 12 10-25-2020 09:19 PM
Wanted: research tool on Android nekokami Android Developer's Corner 2 07-10-2010 10:27 PM
iPad A useful Wikipedia research tool 6charlong Apple Devices 2 04-26-2010 10:29 PM
Anyone want to help me out with a little research? PorterHarbold Lounge 2 10-24-2008 04:52 AM


All times are GMT -4. The time now is 01:34 AM.


MobileRead.com is a privately owned, operated and funded community.