INvestigating and Treebanking Scots And Yiddish Together (INTSAYT)

Treebanks, corpora of syntactically-annotated sentences, have revolutionised the study of language change by providing the data needed for large-scale and precise investigations, combining grammatical analysis with statistical analysis of changes over time. However, while many interesting hypotheses require tens or even hundreds of millions of words to test, the slow speed of manual annotation limits manually-corrected treebanks to at most 1-2 million words. Our project builds treebanks for two underserved minority languages, Yiddish and Scots. We begin by building smaller, manually-corrected treebanks, which are crucial for research into historical linguistics and language variation and change. We then use recent advances in Natural Language Processing (NLP) to automatically annotate texts of much larger size than could be done manually. These larger treebanks will contain c.650 million words for Yiddish and c.13 million words for Scots, both spanning time periods of several hundred years. Building these new treebanks allows for the development and testing of novel hypotheses at a new level of precision, including in the application of information theory to language. First, word order variation in modern and historical Yiddish provides an ideal testing ground for novel information-theoretic hypotheses about syntactic planning. Scots provides another important testing ground for information-theoretic work, but in the domain of morphosyntactic variation. In order to accurately assess information spread and test subtle quantitative hypotheses about how information spread affects the syntactic form of sentences, we need far more examples of word order variation with syntactic annotation marking the subject and objects than can be provided with the smaller manual corpora. Additionally, the larger Scots treebank will allow investigation of socially conditioned variation and change in Scots. By including material from before and during the 17th century decline of Scots features in favour of English, the larger treebank allows exploration into whether Scots syntactic features, possibly less salient than, e.g., orthography in the minds of users, persist in largely English writing. The ability to automatically construct such large treebanks with sufficient accuracy is due to the massive improvements in recent years in the accuracy of syntactic parsers, such as the ones we will train on the manual treebanks to annotate the larger treebanks. Recent work has focused in particular on parsing historical material specifically for purposes of linguistic research. We will take advantage of these developments within the context of a joint effort of linguists who are experts in corpus creation and use, and NLP researchers with knowledge of linguistics. This joint effort will even directly impact the linguistic research itself: the language models developed for the NLP work will themselves form the basis for a new method of assessing sentential information distributions, a novel application of these models to basic science research. Both the Yiddish and Scots texts contain errors resulting from Optical Character Recognition (OCR), along with a great deal of orthographic variation, a situation common to many languages. Our solutions to these issues will therefore be applicable for research in other languages that face similar challenges (e.g., Welsh or Irish). The outcomes of this work will also impact fields outside of corpus-based linguistic research, e.g. historical, literary, and philological studies. Beyond academia, the resources and subsequent research will increase research-informed awareness of Yiddish and Scots as important minority heritage languages, which will aid efforts to preserve and promote these languages and speaker communities.