!!!Meeting setup * Date: 5.11.2007 * Time: 09.30 Norw. time * Place: Internet * Tools: SubEthaEdit, iChat/Skype !!!Agenda # Opening, agenda review # Reviewing the task list from last week # Documentation - divvun.no # Corpus gathering # Corpus infrastructure # Infrastructure # Linguistics # name lexicon infrastructure # Spellers # Other issues # Summary, task lists # Closing !!!1. Opening, agenda review, participants Opened at 10:00. Present: __Børre, Ilona, Per-Eric, Risten, Sjur, Thomas, Tomi, Trond__ Absent: __none__ Agenda accepted as is. !!!2. Updated task status since last meeting !!Børre * move __Steinar's__ error markup in the xml files to (a copy of) the original ** Not done * adding support for the sami languages in OpenOffice.org ** Made some locales * fix Unicode bug in Hunspell conversion java code ** Done some more research, but it still persists * fix [bug 550|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=550] ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** not done * other: ** Have prepared the G5 to take over the role as webserver instead of giellatekno.uit.no. ** Installed Leopard on my machine ** Added derivations to hunspell generation with a lot of help from __Tomi__. The problem with the approach we used is that sme.aff went from 2.4Mb to 34 Mb, so we'll have to come up with a better way of doing it. !!Ilona * lexicalise {{smj}} missing words ** Working on it. * add {{smj}} proper nouns ** No list for it * other {{smj}} tasks ** None !!Maaren * lexicalise actio compounds !!Per-Eric * lexicalise words from the Olavi missing list ** worked and still working * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** nothing done !!Risten * finish the design/text for the CD and the cover ** still working * set up CD-printing printer ** Leif Åge has ordered a new printer !!Saara * add new XSL/XML headers for proofing test docs * Set up ways of adding meta-information for proofing correct corpus docs (source info, used in testing or not, added to lexicon or not) * add nested error markup to xml conversion !!Sjur * work on the XML name editor/risten.no integration ** nope:( * set up risten.no on the G5 again ** added [eXist|http://129.242.220.113:8080/exist/index.xml] (ver. 1.1.1-newcore, actually a very old release, but stable) * test new and nested error markup * get command line hyphenator for automated testing of the hyph-lexicons ** should arrive today * add hyphenation testing ** waiting for the hyphenator * add hunspell testing ** working on it * fix [bug 550|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=550] ** nothing * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** added, closed, commented * other tasks: ** installed Leopard, made [installation notes|/doc/infra/leopard-upgrade.html] ** tested hunspell manually in OOo ** automated hunspell testing, added request for a new output format more suitable for usability testing (suggestion quality) !!Thomas * {{sme->smj}} lexicon conversion to build bilingual lexicon resources ** some done * add {{smj}} proper nouns ** some done * check for bad hyphenation ** not this week * look at test cases still not behaving properly ** some done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** some done !!Tomi * Hunspell lexicon conversion ** worked * {{sme->smj}} lexicon conversion to build bilingual lexicon resources ** not done * fix Unicode bug in Hunspell conversion java code ** it is fixed ? * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** fixed speller bugs !!Trond * {{sme->smj}} lexicon conversion to build bilingual lexicon resources ** Been working hard on this one, progress. * [fix bugs!|http://giellatekno.uit.no/bugzilla]. ** Not in Bugzilla last week :-( !!!3. Documentation Nothing new. __TODO:__ * fix [bug 550|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=550] (__Børre, Sjur__) !!!4. Corpus infrastructure Nothing. !!!5. Infrastructure __Børre__ has been working on moving giellatekno.uit.no and www.divvun.no to the G5. The plan is to finish the work tomorrow. __Sjur__ has installed eXist on the G5 (but no risten.no yet). !!!6. Linguistics !!North Sámi Proper nouns and hyphens: when the name derives, like in -laš, we should get a hyphenation point in front of the derivation, but we don't. Example: {{{ Oslo > os^lo*laš * = missing hyphenation point sme $ lookup -flags mbTT -utf8 bin/hyph-sme.fst 0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100% oslolaš oslolaš Os^lolaš sme$lookup -flags mbTT -utf8 bin/hyphrules-sme.fst 0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 100% oslolaš oslolaš os^lo^laš }}} One possible solutioln is to introduce a marker for derivational suffixes (and, by extension to other languages, also prefixes): * > - suffix * < - prefix What about inflectional suffixes? ''>'' is used in {{sma}} and {{nob}}. This speaks against using it for derivation. {{{ root*1*der*2*infl %> %> Der1 Der2 Der3 C+ VC* CVC l u tags, stem l l diacr here = g u diacr here g l wordform (w or w/o hyphen) }}} >:0 - no hyphenation If visisble hyphenation: >:- IFF _ C V >:0 IFF _ C # {{{ HYPH = read regex ( @\"$(TARGET)/bin/hyphrules-$(TARGET).fst\" .o. \ @\"$(TARGET)/bin/hyph-i$(TARGET).save\" .o. \ @\"$(TARGET)/bin/$(TARGET)-norm.fst\" ) ; \n }}} {{{ SPLRNONREC = read regex [ [@\"$(TARGET)/bin/derivation-filter.fst\" ] .o. \ [@\"$(TARGET)/bin/spellernonrec-$(TARGET).save\" ] .o. \ [@\"common/bin/downcase.fst\" ] .o. \ [@\"common/bin/remove-hyphen.fst\" ] .o. \ [@\"$(TARGET)/bin/hyphrules-$(TARGET).fst\".i ] \ ] ; \n $(TARGET)/bin/spellernonrec-$(TARGET).fst: \ common/bin/tag-not-save.fst \ common/bin/downcase.fst \ $(TARGET)/bin/spellernonrec-$(TARGET).save \ $(TARGET)/bin/derivation-filter.fst \ common/bin/remove-hyphen.fst \ $(TARGET)/bin/hyphrules-$(TARGET).fst os^lolaš os^lolaš Oslo+N+Prop+Plc+Der1+Der/laš+A+Sg+Nom os^lolaš Oslo+N+Prop+Plc+Der1+Der/laš+A+SgGenCmp+Cmpnd os^lolaš Oslo+N+Prop+Plc+Der1+Der/laš+A+SgNomCmp+Cmpnd os^lolaš Oslo+N+Prop+Plc+Der1+Der/laš+A+Attr os^lolaš Oslo+N+Prop+Plc+Der1+Der/laš+A+Attr+Cmpnd os^lo^laš os^lo^laš os^lo^laš +? }}} In {{smj}}, also inflections fall outside our hyphenation rules: {{{ Basudissaj Basudissaj Ba^su^dissaj Bájddárin Bájddárin Bájd^dárin Bájddárin Bájd^dárin Heandarahkii Heandarahkii Hean^da^rahkii }}} __TODO:__ * fix hyphenation of derivations (__Thomas, Tomi, Sjur, Trond__) ** meeting today after lunch, 12:30 !!Lule Sámi __TODO:__ * lexicalise words from the Olavi missing list, but check against the pdf original where in doubt (__Per-Eric__) * {{sme->smj}} lexicon conversion to build bilingual lexicon resources, and increase {{smj}} coverage (__Trond, Thomas, Svenne__) * {{sme->smj}} name conversion (__Trond, Thomas__) * add proper nouns (__Thomas, Ilona__) * look at missing baseforms (__Thomas__) !!!7. Name lexicon infrastructure This sub-project needs to get up and running soon. Mainly __Sjur's__ task. Decisions made in Tromsø can be found in [this meeting memo.|/doc/admin/physical_meetings/tromso-2006-08-propnoun.html] __TODO:__ # set up Tomcat and risten.no on the G5 again (__Sjur, Børre__) ## install risten.no ### eXist is now installed # fix bugs in lexc2xml; add comments to the log element (__Saara__) # finish first version of the editing (__Sjur__) # test editing of the xml files. If ok, then: (__Sjur, Thomas, Trond__) # make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (__Sjur, Saara__) # convert propernoun-($lang)-lex.txt to a derived file from common xml files (__Sjur, Tomi, Saara__) # implement data synchronisation between [risten.no|http://www.risten.no] and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way) # start to use the xml file as source file # clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (__Thomas, Maaren, linguists__) # merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (__linguists__) # publish the name lexicon on risten.no (__Sjur__) # add missing parallel names for placenames (__linguists__) # add informative links between first names like Niillas and Nils (__linguists__) !!!8. Proofing tools !!Hunspell TODO: # Follow-up support for the sami languages in OpenOffice.org (__Børre, Sjur__) # Hunspell lexicon conversion (__Tomi, Børre__) # add hunspell testing to the make file (__Sjur__) # debug and fix remaining conversion issues (__Børre, Tomi__) !!Testing !Spelling Error Markup __TODO:__ * Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) (__Saara__) * move __Steinar's__ error markup in the xml files to (a copy of) the original (__Børre, Kimme__) * add nested error markup to xml conversion (__Saara__) * test new and nested error markup (__Sjur__) !Automated testing __TODO:__ * add hyphenation testing (__Sjur__) * add hunspell testing (__Saara, Sjur__) * add paradigm testing (__Børre, Sjur__) !!Lexicon conversion to the PLX format Open issues based on test results: !smj {{{ svierigaládde svierigaládde svieriga#ládde+N+Sg+Nom polardutkamin polardutkamin polar#dutkat+V+TV+Actio+Der3+Der/m+N+Ess polardutkamin polar#dutkat+V+TV+Actio+Der3+Der/m+N+Sg+Ine svieriga- svieriga- svieriga+Cmpnd polar- polar- polar+Cmpnd }}} 482 - still problematic, 552 - still open, Svierigadárogielan - still rejected !sme 397 - not as good as it should be, 408 - fixed, 419, 425 - roman number, 431, 452 - ''miel'' is a prefix, 461 - ''ovda'' accepted, almost 50 % (17) gets correct suggestion, 489, 508 - fixed, 522, 524. __TODO:__ * look at test cases still not behaving properly (__Thomas, Tomi__) !!InDesign tools __TODO:__ * add hyphenation testing (__Sjur__) !!Hyphenators We should look into the possibility of generating pattern-based hyphenation for OOo. It shouldn't be too hard, or require too much work, but needs investigation. __TODO:__ * get command line hyphenator (__Sjur__) !!Release version The CD cover etc. will be worked on by __John-Marcus Kuhmunen__, and will follow the SD design rules. He is now waiting for the text to be put on the CD cover and other places. There is a bug when installing from CD over an old installation. __TODO:__ * try to burn a CD at SD (__Risten, Leif-Åge__) * write text to go on the CD cover (__Risten__) * set up CD-printing printer (__Risten__) * fix Windows CD installation bug (__Sjur, Børre__) !!Actual release December 11-13, one of these days. !!!9. Other !!Corpus contracts Delayed till after final release. TODO: * publish corpus contracts and project infra as open-source on NoDaLi-sta (__Sjur__) !!Faroese Speller for {{fao}} using our infrastructure and the knowledge we have. __TODO:__ * set up a telephone meeting with them and __Sjur__ (__Trond__) !!Bug fixing When fixing bugs, record the version number containing the fix in the Bugzilla bug report, such that for each bug, we know exactly when it should have been fixed, in what file(s) and what version. __69__ open Divvun/Disamb bugs (__35__ of these 56 are speller-related bugs, __34__ are other bugs), and __23__ risten.no bugs !!SD yearly personell seminar 6.-7. December. __Sjur__ will discuss it with Julia, but our view is that we don't have time to go this year. The release are just a few days later. !!Software updates * SubEthaEdit * Leopard, 10.5 * Skype 2.6.x !!!10. Next meeting, closing The next meeting is 12.11.2007, 09:30 Norwegian time. The meeting was closed at 11:37. !!!Appendix - task lists for the next week !!Boerre [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Boerre.ics] * move __Steinar's__ error markup in the xml files to (a copy of) the original * follow-up support for the sami languages in OpenOffice.org * fix Unicode bug in Hunspell conversion java code * fix [bug 550|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=550] * install Leopard on the G5 and the Xserve * move giellatekno.uit.no (including Bugzilla) and www.divvun.no to the G5. * add paradigm testing * fix Windows CD installation bug * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Ilona [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Ilona.ics] * lexicalise {{smj}} missing words * add {{smj}} proper nouns * other {{smj}} tasks, ask __Thomas__ !!Maaren * lexicalise actio compounds !!Per-Eric [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Per-Eric.ics] * lexicalise words from the Olavi missing list * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Risten [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Risten.ics] * finish the design/text for the CD and the cover * set up CD-printing printer * try to burn a CD at SD !!Saara [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Saara.ics] * add new XSL/XML headers for proofing test docs * Set up ways of adding meta-information for proofing correct corpus docs (source info, used in testing or not, added to lexicon or not) * add nested error markup to xml conversion * add hunspell testing !!Sjur [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Sjur.ics] * work on the XML name editor/risten.no integration * set up risten.no on the G5 again * test new and nested error markup * get command line hyphenator for automated testing of the hyph-lexicons * add hyphenation testing * add hunspell testing * add paradigm testing * fix [bug 550|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=550] * fix hyphenation of derivations, inflections * follow-up support for the sami languages in OpenOffice.org * fix Windows CD installation bug * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Thomas [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Thomas.ics] * {{sme->smj}} lexicon conversion to build bilingual lexicon resources * add {{smj}} proper nouns * check for bad hyphenation * look at test cases still not behaving properly * fix hyphenation of derivations, inflections * {{sme->smj}} name conversion * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Tomi [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Tomi.ics] * Hunspell lexicon conversion * {{sme->smj}} lexicon conversion to build bilingual lexicon resources * fix Unicode bug in Hunspell conversion java code * fix hyphenation of derivations, inflections * [fix bugs!|http://giellatekno.uit.no/bugzilla] !!Trond [iCal|/doc/admin/weekly/2007/Tasks_2007-11-05_Trond.ics] * {{sme->smj}} lexicon conversion to build bilingual lexicon resources * fix hyphenation of derivations, inflections * {{sme->smj}} name conversion * telephone meeting with __Sjur__ and the faroese group re faroese speller * [fix bugs!|http://giellatekno.uit.no/bugzilla].