!!!Meeting setup * Date: 27.8.2007 * Time: 09.30 Norw. time * Place: Internet * Tools: SubEthaEdit, iChat/Skype !!!Agenda # Opening, agenda review # Reviewing the task list from last week # Documentation - divvun.no # Corpus gathering # Corpus infrastructure # Infrastructure # Linguistics # name lexicon infrastructure # Spellers # Other issues # Summary, task lists # Closing !!!1. Opening, agenda review, participants Opened at 09:39. Present: __Børre, Ilona, Sjur, Thomas, Tomi, Trond__ Absent: __Maaren, Per-Eric__ Agenda accepted as is. !!!2. Updated task status since last meeting !!Børre * run all known spelling errors in the prooftest corpus through the speller ** not done * add extraction of all known spelling errors in the regular corpus (not the {{prooftest}} corpus), and check that they are properly corrected ** not done * move __Steinar's__ error markup in the xml files to (a copy of) the original ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** nothing fixed * other ** sent contracts to __Berit Johnskareng__ regarding her Davvi Girji books ** fixed contracts with __Lene Antonsen__ regarding her book "Jámešgušbákti ja Jiehtanas" ** contacted __Johan Jernsletten__ about the Ginna, Gálka, Borta, Snorra books. Will have to contact Sverre Hatle and Karen Anne Oskal Eira, too. ** Began modifying [jazzy-0.52|http://jazzy.sourceforge.net] to include support for sfst transducers. Jazzy is a java implementation of the aspell speller correction algorithm. !!Ilona * lexicalise missing words * add {{sme}} names from FIN ** working on it !!Maaren * lexicalise actio compounds !!Per-Eric * expand the smj typos list * add missing smj words * lexicalise words from the Olavi missing list * add compounding tags to adjectives !!Saara * add new XSL/XML headers for proofing test docs * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Sjur * improve speller test bench: ** run all known spelling errors in the corpus through the speller ** document the AppleScript testing tool ** integrate regression self tests with the make file *** done, very helpful, it has identified a major hole in the PLX conversion ** integrate the ccat speller testing options in the make file * publish corpus contracts and project infra as open-source on NoDaLi-sta ** nothing done * fix stuorra-oslolaš lower case {{o}} ** nothing done * {{ä/æ}} in {{smj}} speller ** compiled new speller - use it as the basis for further testing * restart work on the XML name editor/risten.no integration ** nothing real last week * plan the rest of the project period ** did some, not finished * resend {{smj}} speller-correct document to __Kåre Tjikkom__ ** done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** worked on them, added more reports, and comments on existig ones * other things: ** recompiled spellers ** tested the typos lists on the new spellers !!Thomas * work with compounding ** worked * fix stuorra-oslolaš lower case {{o}} ** not done * {{ä/æ}} in {{smj}} speller ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** not done !!Tomi * make PLX conversion test sample; add conversion testing to the make file ** not done * add Hunspell data generation/conversion ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** fixed !!Trond * Work on the web corpus issues * update the {{smj}} proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt ** Not done. * fix stuorra-oslolaš lower case {{o}} ** Not done. * add {{sma}} texts to the corpus repository ** Not done * {{ä/æ}} in {{smj}} speller ** Not done. * [fix bugs!|http://giellatekno.uit.no/bugzilla]. ** Worked on bugs. !!!3. Documentation Nothing new. !!!4. Corpus gathering __Børre__ has contacted several people, see task status above. __TODO:__ * add {{sma}} texts to the corpus repository (__Trond__) !!!5. Corpus infrastructure __Saara__ has removed the *.html lists from the xdoc folder, and our multilingual analyser interface is generated via xml. !!!6. Infrastructure We are ordering a new server for faster processing. - Order not yet placed, we are waiting for the final, updated and corrected offer. It should arrive today. !!!7. Linguistics !!North Sámi Remaining twol issues: see [bug #460|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=460]. The error at present is connected to forms ''olmmoš–'' and ''háliit'', as before. __Ilona__ is working on the list of {{sme}} names from Finland. The list includes lots of enare sámi names. __Børre__ has sent her a new list that contains all northern sami placenames. __Ilona__ has started with that list instead of the Finland name list. The new list should include all the {{sme}} names, also the names that are in the first list. There is an empty file, {{gt/smn/src/propernoun-smn-lex.txt}}, in cvs. Inari names go there, and may then eventually be exported to other lgs. TODO: * lexicalise actio compounds. Example: ''vuolggasadji'' vs. ''vuolginsadji'' (__Maaren__) * fix stuorra-oslolaš lower case {{o}} (__Sjur, Thomas, Trond__) * fix twol bug (__Sjur, Thomas, Trond__) ** meet online this week - Thursday around 12 AM Norwegian time * add the {{sme}} place names from Finland (__Ilona__) ** working on it !!Lule Sámi The {{æ-ä}} issue: see [bug 411 |http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=411]. TODO: * refine {{smj}} proper noun lexica, cf. the propernoun-smj-lex.txt (__Thomas, Trond__) * {{ä/æ}} in speller (__Tomi, Sjur__) ** Works in transducer, not in speller, see bug report (#411) * lexicalise words from the Olavi missing list, but check against the pdf original where in doubt (__Per-Eric__) * add compounding tags to (some weeks of work): ** nouns (__Thomas__) *** a few weeks left ** adjs (__Per-Eric__) *** almost finished * resend {{smj}} speller-correct document to __Kåre Tjikkom__ (__Sjur__) ** done !!!8. Name lexicon infrastructure This sub-project needs to get up and running soon. Mainly __Sjur's__ task. Decisions made in Tromsø can be found in [this meeting memo.|/admin/physical_meetings/tromso-2006-08-propnoun.html] __TODO:__ # fix bugs in lexc2xml; add comments to the log element (__Saara__) # finish first version of the editing (__Sjur__) # test editing of the xml files. If ok, then: (__Sjur, Thomas, Trond__) # make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (__Sjur, Saara__) # convert propernoun-($lang)-lex.txt to a derived file from common xml files (__Sjur, Tomi, Saara__) # implement data synchronisation between [risten.no|http://www.risten.no] and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way) # start to use the xml file as source file # clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (__Thomas, Maaren, linguists__) # merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (__linguists__) # publish the name lexicon on risten.no (__Sjur__) # add missing parallel names for placenames (__linguists__) # add informative links between first names like Niillas and Nils (__linguists__) !!!9. Spellers !!OOo spellers __Tomi__ is working on the lexicon conversion to the Hunspell format. It is moving forward. {{{ plx source -> transducer -> wordlist in plxformat -> speller binary src/* *-plx.fst > 60 GB 2 MB polderland/* hun source -> transducer -> java/perl-server program -> huncode src/* *-hunspell.fst }}} My question: The set of noun-verb-adj continuation lexica is taken from a wordlist, generated by xfst. # generate full paradigm per word with xfst (as for polderland today) # extract stems automatically <= from the generated paradigm (60 GB) and not from the *-lex.txt files. # turn the result into hunspell stem / cont The hunspell generation process thus mirrors the plx generation process. Yes, but with a different-format output (and a much smaller file - Mb instead of Gb). That's fine. The important point is that it has the same input, so that Tomi does not need to rewrite 8000 lines of smX-lex.txt + twol code. Of course:) We have, in parallel, been looking at sfst. The results are good, sfst seems a good compiler for fst-s. How it behaves when scaled up to real size we do not know. A major point regarding spell checkers will be speed - how fast is the sfst engine? The sfst version of smX will be put on hold to after newyear. TODO: * add Hunspell data conversion (__Tomi__) ** working on it !!Testing !Spelling Error Markup __TODO:__ * Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) (__Saara__) * move __Steinar's__ error markup in the xml files to (a copy of) the original (__Børre__) !Automated testing We need a separate speller pre-processor, to turn ccat output into suitable speller input. This we will need to be able to run whole documents through the speller, to test lexical coverage as well as p/r etc. __TODO:__ * document the AppleScript testing tool (__Sjur__) * improve speller test bench (__Sjur__) ** create a speller preprocessor (__Børre__ or __Sjur__) ** integrate the {{ccat}} speller testing options in the Makefile (__Sjur__) * add extraction of all known spelling errors in the corpus (not the {{prooftest}} corpus), and check that they are properly corrected ** put on hold until we have such markup in the regular corpus * add regression __self-test__: ** integrate these in the make file (__Sjur__) *** done !!Lexicon conversion to the PLX format We have found a [bug|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=484] in the conversion. We also have the [double-hyphen issue|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=458], which needs to be fixed in the LexC code (the R lexicon and surrounding matters). We need a compounding form without a hyphen in speller. In xfst processing (adding PLX-tags) you can add hyphens but we don't know how to remove a hyphen. But is this a lexc problem? Probably not, see below (sme.fst, sme-norm.fst, spellernonrec-sme.save all give the same result): {{{ -bash-3.00$ lookup -flags mbTT -utf8 sme/bin/spellernonrec-sme.save 0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100% biila- biila- biila+N+SgNomCmp+Cmpnd biila-- biila-- biila-- +? -bash-3.00$ lookup -flags mbTT -utf8 sme/bin/spellernonrec-sme.fst 0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100% bii^la bii^la biila+N+Sg+Nom bii^la- bii^la- biila+N+SgNomCmp+Cmpnd bii^la-- bii^la-- bii^la-- +? }}} __TODO:__ * fix bug 484 (__Tomi__) * fix bug 458 (__Trond, Sjur__) !!New public beta Delayed till the majority of the present bugs are fixed. The twolc bug is the major stopper. !!!10. Other !!Corpus contracts TODO: * publish corpus contracts and project infra as open-source on NoDaLi-sta (__Sjur__) !!Bug fixing When fixing bugs, record the version number containing the fix in the Bugzilla bug report, such that for each bug, we know exactly when it should have been fixed, in what file(s) and what version. __57__ open Divvun/Disamb bugs (__29__ of these 56 are speller-related bugs, __28__ are general bugs), and __23__ risten.no bugs !!Project meeting We'll meet in September, 24-28, in Tromsø to work on the hardest remaining issues. !!!11. Next meeting, closing The next meeting is 3.9.2007, 09:30 Norwegian time. The meeting was closed at 10:54. !!!Appendix - task lists for the next week !!Boerre * run all known spelling errors in the prooftest corpus through the speller * add extraction of all known spelling errors in the regular corpus (not the {{prooftest}} corpus), and check that they are properly corrected * move __Steinar's__ error markup in the xml files to (a copy of) the original * create a speller preprocessor * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Ilona * lexicalise missing words * add {{sme}} names from FIN * make {{smn}} propernoun-list !!Maaren * lexicalise actio compounds !!Per-Eric * expand the smj typos list * add missing smj words * lexicalise words from the Olavi missing list * add compounding tags to adjectives !!Saara * add new XSL/XML headers for proofing test docs * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Sjur * improve speller test bench: ** document the AppleScript testing tool ** create a speller preprocessor ** integrate the ccat speller testing options in the make file * publish corpus contracts and project infra as open-source on NoDaLi-sta * fix stuorra-oslolaš lower case {{o}} * {{ä/æ}} in {{smj}} speller * work on the XML name editor/risten.no integration * plan the rest of the project period * fix {{sme}} twol bug (#460), meeting Thursday at 12 AM * fix bug 458 * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Thomas * work with compounding * fix stuorra-oslolaš lower case {{o}} * {{ä/æ}} in {{smj}} speller * fix {{sme}} twol bug (#460), meeting Thursday at 12 AM * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Tomi * make PLX conversion test sample; add conversion testing to the make file * add Hunspell data generation/conversion * fix bug 484 * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Trond * update the {{smj}} proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt * fix stuorra-oslolaš lower case {{o}} * add {{sma}} texts to the corpus repository * {{ä/æ}} in {{smj}} speller * fix {{sme}} twol bug (#460), meeting Thursday at 12 AM * fix bug 458 * [fix bugs!|http://giellatekno.uit.no/bugzilla].