Nepali National Corpus

The core corpus is a collection of Nepali written texts that concur as far as possible with the date, number and genres of the international FLOB and FROWN corpora consisting of 500 texts of 15 different genres with 2000 words each published between 1990 and 1992. This framework is as follows:

Table 1: Core sample framework (based on FLOB/FROWN corpora)
Category Label	Category Title	Number of samples
A	Press: Reportage	44
B	Press: Editorial	27
C	Press: Review	17
D	Religion	17
E	Skills, Trades and Hobbies	38
F	Popular Lore	44
G	Belles Lettres, Biographies, Essays	77
H	Miscellaneous	30
I	Science	80
J	General Fiction	29
K	Mystery and Detective Fiction	24
L	Science fiction	6
M	Adventure and Western	29
N	Romance and Love story	29
O	Humour	9
	TOTAL	500

The primary purpose of the Core Sample was to provide a match to other corpora created from the same sampling frame. However, there were made some adaptations for selecting genres as all the genres existing in English writings (e.g. science fiction) did not exist in Nepali because of cultural and other East-West differences. Besides, only 398 (instead of 500) texts could be collected for Nepali core corpus since texts from some genres could not be available from the 1991/92 time frame when writings in Nepali were very much restricted and  just started broadening with the advent of liberalism after the restoration of democracy in the country.

These collected core corpus is presented in Table 2.

Table 1: Core sample framework (based on FLOB/FROWN corpora)
Category name	No of files	No of words
A (Press reportage)	33 (44)	66800
B (Press editorial)	23 (27)	46520
C (Press review)	6 (17)	12095
D (Religion)	13 (17)	26412
E (Skills, Trades and Hobbies)	29 (38)	58935
F (Popular lore)	32 (44)	64878
G (Belles Letters, Biographies, Essays)	68 (77)	137873
H (Miscellaneous)	28 (30)	56680
J (Science)	56 (80)	113507
S (Fiction)	110 (126)	220874
Grand total	398 (500)	804574

The internal structure of the core corpus is as follows:

Press editorial
Daily:		5
From Kathmandu:	6
From Outside:	–
Weekly:		17
From Kathmandu	15
From Outside:	–
Half–weekly:		1
From Kathmandu:	–
From Outside:	1
Total		23

Religion
From Book	10
One text translated from Hindi and one text based on Sanskrit
From article:	3
Total	13
Fiction
Novel	66
Short story	44
Total	110
Science
From book	37
From periodicals	19
Total	56
Sub-category
Science and technology	3
Criticism	20
Anthropology / culture	8
History / Archeology	5
Language and grammar	3
Law / politics	5
Psychology	1
Philosophy	4
Business / economics / administration	6
Unclassified	1

These 1 million words appearing in 398 texts extracted from various books, journals, magazines and newspapers were digitized in Nepali Unicode. For the purpose of computer processing, these texts were then manually formatted using XML tagging in the body, paragraph, sentences and foreign words appearing therein. Each text was provided with the metadata or  bibliographical details such as book/article/ issue title, author, publisher, publication date, publication place, name of the typist, etc. in XML header.  Additional relevant XML tags were also added automatically.

A set of 112 parts-of-speech (POS) tags were developed empirically to annotate the core corpus (For details see POS Tagset). This tagset was first manually used to annotate 160 files in the core corpus. Based on this manually tagged corpus, an automatic tagger was developed at LU called Unitag, and has been used to automatically tag the whole of the text corpus using lexicon, rules and probalistic generalizations. However, in line with our policy of technology transfer we have been building our own parts-of-speech tagger at MPP as part of a general morphology analyser for a range of uses.