Bibtex

[ProfElm]

Hello everyone,

I was wondering if someone can take a look at the documentation so far? The link is in my previous post but it can also be read here.

I am not so sure how the .xml file format may be useful when creating a catalog. Any advice would be helpful.

Thanks a lot everyone!

[Debby]

ProfElm, great start! I read through the doc and would like to send you my comments. Would you please send me an email address where I can send the doc back to you?

Manichean - thank you! I have not heard of Mendeley but it looks great! I will download and investigate. Initial impressions?

[Manichean]

Originally Posted by Debby

Manichean - thank you! I have not heard of Mendeley but it looks great! I will download and investigate. Initial impressions?

It does the job. Metadata recognition, even from PDFs, is unexpectedly solid. I mostly use it to avoid having to manually type up BibTeX files, so I can't tell you much about features other than adding files, categorizing them and editing the metadata (only needed in 2 cases with my current library), and then exporting the relevant categories as BibTeX.

Edit for an update: I just dumped about 10 papers that I collected yesterday into my library. Of those, about half got the metadata wrong, although I had to manually edit just two- the rest had a correct title, which enables one to use Google Scholar lookup.

[ProfElm]

Debby, that would be excellent!

Please send the revisions to hrios10 at gmail dot com.

I wonder how Mendeley gets the metadata straight from a document. I must inquire!

[Manichean]

Originally Posted by ProfElm

I wonder how Mendeley gets the metadata straight from a document. I must inquire!

My guess would be internal metadata. I used the papers' title as file name, if it had used the filename, it should've gotten the title correct in every case, which did not happen.

[Debby]

Just sent it over. Let me know if you don't receive it.

D

[ProfElm]

Debbie, thank you so much! I changed a few things that you recommended and tweaked some minor things.

My main question now is: What should go under the .xml files section?

If we can get a way to parse through a documents first few pages for the isbn, that would be epic.

I have also added screenshots and completed the .xml section. Is there anything else that should be added? When can this be incorporated into the manual?

[lidans]

Originally Posted by kovidgoyal

You can already export your calibre catalog in Bintext format. click the arrow next to convert and choose create catalog. Select bibtext as your catalog format.

This works perfectly, thank you execpt that when a book is in Hebrew the convert "losses" the path of the book and (jabref in my try) can not find the book directory

[erymaxuk]

Originally Posted by ProfElm

Debbie, thank you so much! I changed a few things that you recommended and tweaked some minor things.

My main question now is: What should go under the .xml files section?

If we can get a way to parse through a documents first few pages for the isbn, that would be epic.

I have also added screenshots and completed the .xml section. Is there anything else that should be added? When can this be incorporated into the manual?

If you are collecting academic articles, you should be aware of the new standard of publishing articles in xml formate. The NLM and PubMed has millions of articles in DTD JATS xml formate. They have to be transformed with xslt processor such as Saxon ( I use it). The xml file can be transformed in bib, html or other format. I have many articles and books using mssql database. Calibre is an excellent database for eboks, pdf and other formats but you cannot import JATS xml files into calibre. My process flow is following:
download DTD xml files from NLM ftp sites daily. Transform them with Saxon to bib files using jatstobib.xslt. Import all (ususaly 15000) into Mendeley which will update the metadata. Export them as one bib file which I import in Zotero. From there export as csv file and load in excel. The excel file can be easily imported into mssql database and process with Visual studio. (Window Forms in C#)

I also download books and articles as pdf. Import them into Zotero and update metadata, export as csv file, import with zotero importer plugin into calibre. Create in calibre searchable tags and covers. Creating catalog using the command line as csv and load it in excel (same way to mssql).

If you want to know more please contact me.

[DaltonST]

Originally Posted by erymaxuk

If you are collecting academic articles, you should be aware of the new standard of publishing articles in xml formate. The NLM and PubMed has millions of articles in DTD JATS xml formate. They have to be transformed with xslt processor such as Saxon ( I use it). The xml file can be transformed in bib, html or other format. I have many articles and books using mssql database. Calibre is an excellent database for eboks, pdf and other formats but you cannot import JATS xml files into calibre. My process flow is following:
download DTD xml files from NLM ftp sites daily. Transform them with Saxon to bib files using jatstobib.xslt. Import all (ususaly 15000) into Mendeley which will update the metadata. Export them as one bib file which I import in Zotero. From there export as csv file and load in excel. The excel file can be easily imported into mssql database and process with Visual studio. (Window Forms in C#)

I also download books and articles as pdf. Import them into Zotero and update metadata, export as csv file, import with zotero importer plugin into calibre. Create in calibre searchable tags and covers. Creating catalog using the command line as csv and load it in excel (same way to mssql).

If you use Zotero and/or RIS files and/or (soon) BIB files, see the attached image.




DaltonST

[Fiat_Lux]

Originally Posted by erymaxuk

I also download books and articles as pdf. Import them into Zotero and update metadata, export as csv file, import with zotero importer plugin into calibre. Create in calibre searchable tags and covers. Creating catalog using the command line as csv and load it in excel (same way to mssql).

I've tried using Zotero both on its own, and as a precursor to Calibre import, BibText export. From my perspective, the only way for Zotero to have accurate data, is for me to manually key in _all_ of the data fields --- it can't even get author and title correct. The failure isn't swapped data, but rather utter garbage.

I see neither increased bibliographic accuracy, nor time saving in using Zotero.

Does your usage of Zotero result in either saving time, or increased bibliographic accuracy of the material? If so, how?

[erymaxuk]

Originally Posted by Fiat_Lux

I've tried using Zotero both on its own, and as a precursor to Calibre import, BibText export. From my perspective, the only way for Zotero to have accurate data, is for me to manually key in _all_ of the data fields --- it can't even get author and title correct. The failure isn't swapped data, but rather utter garbage.

I see neither increased bibliographic accuracy, nor time saving in using Zotero.

Does your usage of Zotero result in either saving time, or increased bibliographic accuracy of the material? If so, how?

pdf articles drag and drop into Zotero collection, will automaticaly update metadata from pdf files with bibliographic accuracy. Occasionaly some pdf fails, simple open it in Zotero and click on doi URL, this wil open the article in browser, click on Zotero importer and the correct metadata will be downaloded into the collection. In the Zotero folder - in translators folder find csv.js, open it in any notepad++ and you can change setting for example I changed exportNotes to true, to export abstracts.

After all set, export it as csv (not bib), import it into calibre with Zotero importer. After I select all and create covers and tags. Zotero importer is an excellet plug-in, respect for its creator.

In regard of books (pdf, epub ets) is a diferrent story. Import them directly into calibre. The ISBN of books is the important book ID, for articles is the DOI importat ID, which you get easily updated in Zotero wit DOI plugin.

For me the way after is export books or articles from calibre in csv file which I load in excel, make some fine tunes like column formats to change dates to datem numbers to number etc. There is one problem with the filename path for pdf in calibre. It comes in format like this PDF:c:\books\..., I remove the PDF: or EPUB: part in excel before saving the loaded csv as excel file.
In case of different categories of medical books I created a simple VB macro scipt in excel and run it as bat file. I use mostly batch files for calibre on command line; import books into calibre, export as catalog in csv etc.

Excel file can be easily then imorted in MSSQL database in Visual Studio.

[erymaxuk]

Originally Posted by erymaxuk

pdf articles drag and drop into Zotero collection, will automaticaly update metadata from pdf files with bibliographic accuracy. Occasionaly some pdf fails, simple open it in Zotero and click on doi URL, this wil open the article in browser, click on Zotero importer and the correct metadata will be downaloded into the collection. In the Zotero folder - in translators folder find csv.js, open it in any notepad++ and you can change setting for example I changed exportNotes to true, to export abstracts.

After all set, export it as csv (not bib), import it into calibre with Zotero importer. After I select all and create covers and tags. Zotero importer is an excellet plug-in, respect for its creator.

In regard of books (pdf, epub ets) is a diferrent story. Import them directly into calibre. The ISBN of books is the important book ID, for articles is the DOI importat ID, which you get easily updated in Zotero wit DOI plugin.

For me the way after is export books or articles from calibre in csv file which I load in excel, make some fine tunes like column formats to change dates to datem numbers to number etc. There is one problem with the filename path for pdf in calibre. It comes in format like this PDF:c:\books\..., I remove the PDF: or EPUB: part in excel before saving the loaded csv as excel file.
In case of different categories of medical books I created a simple VB macro scipt in excel and run it as bat file. I use mostly batch files for calibre on command line; import books into calibre, export as catalog in csv etc.

Excel file can be easily then imorted in MSSQL database in Visual Studio.

In regard of collecting only the citations, as I mention above, I download the DTD JATS xml files from any source like NLM, HINDAWI od MDPI normaly from their FTP sites. Those xml file need to be transormed using xslt in Saxon or other processors. I use Saxon HE 12 with a powershell script to collect thousands of xml files to transform in bib files. You can also transform in html, csv files. I load the bib files in Mendeley, update metadata and export thousands of bib files in one bib file. Import the bib files into Zotero, update doi and export as csv -> excel - >sql databse

I posted before screenshots of my finished forms created in visual studio in which I access my database with books, articles and citations. Also full journal collections.

[erymaxuk]

Originally Posted by erymaxuk

In regard of collecting only the citations, as I mention above, I download the DTD JATS xml files from any source like NLM, HINDAWI od MDPI normaly from their FTP sites. Those xml file need to be transormed using xslt in Saxon or other processors. I use Saxon HE 12 with a powershell script to collect thousands of xml files to transform in bib files. You can also transform in html, csv files. I load the bib files in Mendeley, update metadata and export thousands of bib files in one bib file. Import the bib files into Zotero, update doi and export as csv -> excel - >sql databse

I posted before screenshots of my finished forms created in visual studio in which I access my database with books, articles and citations. Also full journal collections.

some scrshots of my collections

[Fiat_Lux]

Originally Posted by erymaxuk

pdf articles drag and drop into Zotero collection, will automatically update metadata from pdf files with bibliographic accuracy. Occasionally some pdf fails, simple open it in Zotero and click on doi URL, this wil open the article in browser, click on Zotero importer and the correct metadata will be downloaded into the collection.

So Zotero works if, and only if either of the following is true:
* The PDF contains accurate metadata;
* The PDF has a DOI printed somewhere within;

I suspected that something like that was the case.