I have for now posted at en.wp, https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Automatic_citations_based_on_ISBN_are_broken .

Is this a security issue? Has anyone contacted WorldCat to find out why our API access stopped working? According to pages like https://help.oclc.org/Librarian_Toolbox/OCLC_APIs/Troubleshooting/How_do_I_request_an_API_WSKey and https://www.oclc.org/en/partnerships.html#section-apiservices the API seems to be available, including to organizations.

In T336298#8841082, @Jonesey95 wrote:

Is this a security issue? Has anyone contacted WorldCat to find out why our API access stopped working? According to pages like https://help.oclc.org/Librarian_Toolbox/OCLC_APIs/Troubleshooting/How_do_I_request_an_API_WSKey and https://www.oclc.org/en/partnerships.html#section-apiservices the API seems to be available, including to organizations.

Please see my comment at T336297#8842993 - we know what the issue is, it's just going to take some effort to resolve.

I just sent the Mass message, please LMK if there's any problem with it. I'd like to publicly thank my many colleagues at WMF who are working hard to resolve this issue, and I ask for your patience and understanding on the matter. I'll make sure that this task is updated whenever we have anything worth sharing.

Stupid question. Is the Open Library API https://openlibrary.org/dev/docs/api/books a usable alternative? In particular ISBN via the .json format?

In T336298#8846535, @DeirgeDel wrote:

Stupid question. Is the Open Library API https://openlibrary.org/dev/docs/api/books a usable alternative? In particular ISBN via the .json format?

I presume that Open Library is not as comprehensive (it's the same as any other downstream collection like Google Books, it doesn't contain all possible ISBNs).

Confirmed w/ Mek @ IA that there should be no problem hitting their /isbn endpoint at 5rps.
We may want to start with OL as the primary resolver, and only fall back to Worldcat if OL doesn't have full metadata for that ISBN - especially if WC continues with its quota system.

In T336298#8849499, @EpicPupper wrote:

In T336298#8846535, @DeirgeDel wrote:

Stupid question. Is the Open Library API https://openlibrary.org/dev/docs/api/books a usable alternative? In particular ISBN via the .json format?

I presume that Open Library is not as comprehensive (it's the same as any other downstream collection like Google Books, it doesn't contain all possible ISBNs).

WorldCat also doesn't contain all possible books. For example many Polish books are not there (or was not there last time I checked).

In OpenLibrary you can also add a book if it is missing so that is a bit better.

Note that OL can be integrated into e.g. refToolbar. It does allow CORS, so it can be used directly (without proxy). You neeed two API calls because the author is not provided in ISBN call.

var re = await fetch("https://openlibrary.org/isbn/9780140328721.json", {
	"credentials": "omit",
	"method": "GET",
	"mode": "cors"
});
var json = await re.json();
console.log(json);
// 1st author
var authorKey = json.authors[0].key;
var author = await fetch('https://openlibrary.org/authors/OL34184A.json', {
	"credentials": "omit",
	"method": "GET",
	"mode": "cors"
})
.then(response => response.json())
;
console.log(author);

I've written a tool for OL itself, but it can be changed to support refToolbar or a standalone on-wiki gadget. See load function here:
https://github.com/Eccenux/wiki-CytujOpenLibrary/blob/master/CytujOpenLibrary.user.js#L18

Instead of readAuthors() call you just need to do above, extra call.

In T336298#8849499, @EpicPupper wrote:

In T336298#8846535, @DeirgeDel wrote:

Stupid question. Is the Open Library API https://openlibrary.org/dev/docs/api/books a usable alternative? In particular ISBN via the .json format?

I presume that Open Library is not as comprehensive (it's the same as any other downstream collection like Google Books, it doesn't contain all possible ISBNs).

In T336298#8849508, @Sj wrote:

Confirmed w/ Mek @ IA that there should be no problem hitting their /isbn endpoint at 5rps.
We may want to start with OL as the primary resolver, and only fall back to Worldcat if OL doesn't have full metadata for that ISBN - especially if WC continues with its quota system.

This is a great suggestion - we have now enabled Zotero's isbn endpoint internally, which uses Library of Congress, BnF ISBN (French library), and K10plus ISBN (German repository) (T336727). I'm not sure how IA compares to LoC in terms of coverage? Definitely worth looking into. I know there is an IA translator in Zotero but not for the isbn endpoint.

I've been using Zotero for managing my personal library extensively over the years. It always used to be superior to the solution we had in Citoid. So I would second using Zotero's ISBN endpoint in Citoid. The Internet Archive cannot be better than that because the national libraries' catalogues are the reference for metadata of any book in print. I'm looking forward to the improvements you are about to implement!

Zotero seems ideal. Thank you!

@Mvolz @VPuffetMichel Please let us know if this should be resolved just by adding a note about it in Tech News.

After some testing in my fields of interest I just would like to thank you for this major improvement in Citoid! In fact, it's amazing, much better that what we've had before. :)

Unfortunately, as for now, many books issued in Poland are not covered. However, @Nux informed me that he had already added the Polish 'National Library' database, which, he hopes, will be included in the next implementation.

I presume that Open Library is not as comprehensive (it's the same as any other downstream collection like Google Books, it doesn't contain all possible ISBNs).

Not a safe assumption. I don't know what you mean by "downstream" or "all possible", but OpenLibrary attempts to import all known ISBN codes with sufficient metadata. Some of their sources can be seen at:

• https://github.com/internetarchive/openlibrary/blob/master/openlibrary/templates/history/sources.html
• https://archive.org/details/ol_data

ISBNdb is more comprehensive, but their licensing is less clear.

I'm not sure how IA compares to LoC in terms of coverage?

LoC dumps get imported somewhat regularly, I'm not sure about updates between dump releases. It's possible to inspect the dumps to compare: https://openlibrary.org/developers/dumps

So I would second using Zotero's ISBN endpoint in Citoid. The Internet Archive cannot be better than that because the national libraries' catalogues are the reference for metadata of any book in print.

Yes and no. Zotero is great and we should use it wherever possible, but it's impractical to run it through hundreds of targets. So it would be far easier to target a meta-catalogue.

Not all national libraries are tasked with cataloguing the entire national book production, and those who do tend to be more active in the open data field, which means their data is more likely to be in OpenLibrary.

K10plus is amazing but it probably skews towards books published in USA/Europe. OpenLibrary can be a good default to cover anything else which we don't know where to search.

Well, not exactly. A meta catalogue is always inferior to the original source because information gets lost. Citoid never worked fine as long as we had only WorldCat there. No objections to additional sources, but please leave K1plus, as is now.

In T336298#8890900, @Nemo_bis wrote:

I presume that Open Library is not as comprehensive (it's the same as any other downstream collection like Google Books, it doesn't contain all possible ISBNs).

Not a safe assumption. I don't know what you mean by "downstream" or "all possible", but OpenLibrary attempts to import all known ISBN codes with sufficient metadata. Some of their sources can be seen at:

• https://github.com/internetarchive/openlibrary/blob/master/openlibrary/templates/history/sources.html
• https://archive.org/details/ol_data

ISBNdb is more comprehensive, but their licensing is less clear.

I'm not sure how IA compares to LoC in terms of coverage?

LoC dumps get imported somewhat regularly, I'm not sure about updates between dump releases. It's possible to inspect the dumps to compare: https://openlibrary.org/developers/dumps

So I would second using Zotero's ISBN endpoint in Citoid. The Internet Archive cannot be better than that because the national libraries' catalogues are the reference for metadata of any book in print.

Yes and no. Zotero is great and we should use it wherever possible, but it's impractical to run it through hundreds of targets. So it would be far easier to target a meta-catalogue.

Not all national libraries are tasked with cataloguing the entire national book production, and those who do tend to be more active in the open data field, which means their data is more likely to be in OpenLibrary.

K10plus is amazing but it probably skews towards books published in USA/Europe. OpenLibrary can be a good default to cover anything else which we don't know where to search.

In T336298#8891183, @Aschmidt wrote:

Well, not exactly. A meta catalogue is always inferior to the original source because information gets lost. Citoid never worked fine as long as we had only WorldCat there. No objections to additional sources, but please leave K1plus, as is now.

I think our policy is definitely going to be to keep the existing translators that Zotero offers (primarily at this point because they understand their codebase best).

There is some comment from the Zotero developers on whether and how many national libraries to target in the PR for the Poland translator @Nux wrote here: https://github.com/zotero/translators/pull/3036

I think the consensus there seems to be that be filtering by the isbn prefix (which has location information) is a good way to enable a proliferation of more translators for the time being.

Targeting of more meta catalogues would probably be something we'd want to discuss with them too, but to me it seems like a good idea to me as a fall back option for where we don't have a national library data, because there will be ones we miss.

There is also the possibility of starting to store some of this data on wikidata and making it a meta catalogue... which right now doesn't really have many books. This would enable us to enhance citations with wikilinks to the authors or the book itself on the wiki, but that'd be pretty pie in the sky at this point.

This ticket is probably the most appropriate place to continue this -> T212585

@Mvolz I think we're technically done with the scope of this task? Thanks.