Methods and Results

Limits before results

A figure without its method is an assertion, not a result. The Initiative therefore publishes what its tools cannot do before it reports what they achieve. This follows one of the values it works by: methodological modesty (التواضع المنهجي).

For this reason no figure appears on this site on its own. Each one is shown with an evidence card. When a value changes, it receives a new card, and the earlier card stays visible with its date. Every card states the following:

What a methods report contains

Every result rests on a methods report. A report is written so that someone outside the Initiative can follow the measurement and question it. It contains at least:

1. The measurement run: its identifier and the versions of the data and software it used.
2. The date: when the measurement was taken and which state of the sources it reflects.
3. The sample: how it was drawn, how large it is, and how the reference judgements used for comparison were made.
4. The procedure: every step, and the definition of each measure, for example precision and recall.
5. Negative tests: deliberately faulty inputs that a check must reject, such as an altered quotation. They show that a check is able to fail at all.
6. Known limits: what the measurement does not show, and where the procedure is known to go wrong.
7. Recalculation: the steps and the script by which the figures can be recalculated. Source data are shared only as far as their licences allow.

Like everything the Initiative publishes, a report appears only after review by people with the relevant expertise, including specialists in philology. AI does not replace scholarship.

Current status

In progress The first methods report, covering the manuscript metasearch and the linking of name forms to persons, is being prepared.

Until this report is published, the site gives no figures on the size of the index or on the accuracy of the tools. Readings of the index size taken on different days have not matched, and the report is to name the cause.

The report is intended to cover two measures: the share of manuscript records with a valid link back to the source at the holding institution, and the precision and recall of name linking.

Planned A documented citation check on the pilot corpus of the text tool, showing accepted and rejected answers together with the negative tests.

Planned An external review of published figures by an independent specialist or institution.

Known limits today

These limits apply now, before any report is published. They are stated here so that no result is read more broadly than it can bear.

• Handwriting recognition: automatic reading of handwritten text in historical Arabic-script holdings is not a solved problem; research projects such as OpenITI AOCP are working on it. The Initiative makes no promises in this area. Its work rests on catalogue records and on texts that have already been transcribed.
• A selection of texts: the text index currently covers a selection of works, not the whole corpus. The text tool cannot check a quotation against a work that is not in the index.
• No copies of digitised manuscripts: images are never mirrored. They stay with the holding institution and are reached through a link back to the source. Whether an image can be viewed depends on that institution, and a link can break if its address changes.
• Catalogues, not collections: the metasearch finds what the connected catalogues describe. A missing record says nothing about whether a manuscript exists.
• Negative findings: if a passage is not found, that means only that it was not found in the given edition at the given place. It is never a judgement on the authenticity of a text.
• Wording, not interpretation: a verified quotation shows that the words stand in the source. It does not show that the answer around it reads the source correctly.
• Names: no two names are merged silently, and every link between name forms states its procedure and its confidence. A link that cannot meet this requirement is not made; the report is to show what that means for the results.

Correction log

Every correction to a figure, a statement of method or a published text is entered here, with its date, what was wrong, what was changed and why. The earlier version remains traceable.

There are no entries yet. No figures or reports have been published so far, so nothing has needed correcting.

Reporting an error

If you find an error in a figure, a method or a text, please let us know through the Contact page. Corrections are entered in the log above, in keeping with the transparency about errors set out in the Charter.

Tools Arabic and Islamic Cultural Heritage Arabic Language and AI