!!!Meeting setup * Date: 23.04.2007 * Time: 10.30 Norw. time * Place: Internet * Tools: SubEthaEdit, iChat !!!Agenda # Opening, agenda review # Reviewing the task list from last week # Documentation - divvun.no # Corpus gathering # Corpus infrastructure # Infrastructure # Linguistics # name lexicon infrastructure # Spellers # Other issues # Summary, task lists # Closing !!!1. Opening, agenda review, participants Opened at 10:51. Present: __Børre, Maaren, Per-Eric, Sjur, Steinar, Thomas, Tomi__ Absent: __Saara, Trond__ Agenda accepted as is. !!!2. Updated task status since last meeting !! Børre * update and fix our documentation and infrastructure as __Steinar__ finds problem areas - low priority ** not done * find missing {{nob}} parallel texts in corpus ** not done * improve number PLX conversion ** not done * add {{sma}} texts to the corpus repository ** not done * collect a list of PR recipients, forward to Berit Karen Paulsen ** not done * run all known spelling errors in the corpus through the speller ** not done * add extraction of all known spelling errors in the regular corpus (not the {{prooftest}} corpus), and check that they are properly corrected ** not done * test speller installers on Windows and Mac ** worked perfectly straight from Polderland, we will have to add our own updates to the spellers and test the installation again * set up Per-Eric's computer ** done * install all required software on Per-Eric's computer ** done * update installer packages with latest speller lexicon ** not done * add numbers as figures to the PLX sources ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** not done * other: ** helped out on the issues with the compilation of the spellers ** added documents to the prooftest folder on victorio !! Maaren * lexicalise actio compounds * Manually mark speller test documents for typos !! Saara * prepare more files for manual alignment ** done * improve cgi-bin scripts ** add new features to the paradigm generator *** not done * add new XSL/XML headers for proofing test docs ** discussion started in newsgrp * compilation of verb lists ** not done since there are some problems * speed in smj conversion ** not done, is this fixed? * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * finish press release for the beta ** not done * collect a list of PR recipients ** not done * make the version info date stamp reflect the compilation date automatically ** done * run all known spelling errors in the corpus through the speller ** not yet done, but the infra is about to become ready * document the AppleScript testing tool ** not finished * integrate regression self tests with the make file ** not yet done, but the infra is ready * make improved {{smj}} speller (incl. derivations and compounds) ** worked on it all week last week * set up Per-Eric's computer ** __Børre__ did it * install all required software on Per-Eric's computer ** __Børre__ did it * set up all user accounts for Per-Eric ** __Børre__ did it * improve speller test bench ** done some, needs more integration * add names to {{smj}} speller ** done * update installer packages with latest speller lexicon ** studied the packages, not yet finished. * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Steinar * Beta testing: Align manually (shorter texts) * Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue ** started working, some text are now added * Complete the semantic sets in sme-dis.rle ** no work this week * missing lists ** added some missing words lately * Look at the actio compound issue when adding from missing lists ** not done * Align corpus manually * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas * work with compounding ** worked and still working * Lack of lowering before hyphen: Twol rewrite. ** not done * translate beta release docs to {{sme}} and {{smj}} ** not done * Add potential speller test texts ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** not any this week !! Tomi * make improved {{smj}} speller (incl. derivations and compounds) ** made, but not finished * make PLX conversion test sample; add conversion testing to the make file ** not done * improve number PLX conversion ** improved a bit * improve prefix and middle-noun PLX conversion ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond * Test the beta versions ** Testing in progress * Work on the parallel corpus issues ** Discuss with Lars *** Done. ** Work on the aligner with (__Børre__) *** Done, good progress on anchor.txt ** fix {{sme}} texts in corpus this month *** Not enough progress here. ** find missing {{nob}} parallel texts in corpus, go through Saara's list *** Included missing school history texts * Postpone these tasks to after the beta: ** update the {{smj}} proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt *** Very postponed. ** Go through the Num bugs *** Not done. * Improve automatic alignment process ** Improved indeed, due to anchor.txt and improved length measure (1.15). * Align corpus manually ** Done. * Add potential speller test texts ** Börre did this * collect a list of PR recipients ** Not looked at that issue. * set up all user accounts for Per-Eric ** Done. * [fix bugs!|http://giellatekno.uit.no/bugzilla]. !!!3. Documentation The open documentation issues fall into these three categories: * Beta documentation for testers * Documentation for the online corpora * General documentation improvement after Steinar's test (for open-source release) TODO: * write form to request corpus user account (__Børre, Sjur, Trond__) ** delayed till after the beta release * document how to apply for access to closed corpus, and details on the corpus and its use in general (__Børre, Sjur, Trond__) ** delayed till after the beta release * correct and improve it based on feedback from __Steinar__ (__Børre__) ** low priority * beta documentation (see separate beta section below) !!!4. Corpus gathering __Børre__ has gathered a lot of documents from the OAO department in Sámediggi, most available in victorio, in our prooftest corpus branch. TODO: * {{sme}} texts: no new additions, fix corpus errors during this month (__Børre, Trond, Saara__) * missing {{nob}} parallel texts should be added if such holes are found (__Børre, Trond__) * Go through the list of missing or errouneous {{nob}} texts, based upon __Saara's__ perfect list (__Børre, Trond__) * add {{sma}} texts to the corpus repository (__Børre__) !!!5. Corpus infrastructure !!Alignment __TODO__ * Improve the automatic process: ** Improve the anchor list and realign (__Trond, Børre__) ** Only adding words does not improve alignment, you have to consider the format as well. If you cut with star e.g guo@ too early, wrong word can be selected. ** The documents have still some formatting issues which cause trouble in alignment. (In some documents tables are included to the text, some not, etc.) ** Test and improve settings in the aligner * Align manually (__Trond, Steinar__) (especially shorter terminological texts) good idea. __Saara__ will look for troublesome texts and prepare them for manual alignment. !!!6. Infrastructure __TODO:__ * update and fix our documentation and infrastructure as __Steinar__ finds problem areas (__Børre__) !!!7. Linguistics !!North Sámi TODO: * lexicalise actio compounds. Example: ''vuolggasadji'' vs. ''vuolginsadji'' (__Maaren__) * fix stuorra-oslolaš lower case {{o}} (__Sjur, Thomas, Trond__) ** postponed till after the public beta !!Lule Sámi A lot of work was done last week, as part of __Thomas__ training __Per-Eric__. TODO: * refine {{smj}} proper noun lexica, cf. the propernoun-smj-lex.txt (__Thomas, Trond__) !!!8. Name lexicon infrastructure Decisions made in Tromsø can be found in [this meeting memo.|/doc/admin/physical_meetings/tromso-2006-08-propnoun.html] __TODO:__ # fix bugs in lexc2xml; add comments to the log element (__Saara__) # finish first version of the editing (__Sjur__) # test editing of the xml files. If ok, then: (__Sjur, Thomas, Trond__) # make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (__Sjur, Saara__) # convert propernoun-($lang)-lex.txt to a derived file from common xml files (__Sjur, Tomi, Saara__) # implement data synchronisation between [risten.no|http://www.risten.no] and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way) # start to use the xml file as source file # clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (__Thomas, Maaren, linguists__) # merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (__linguists__) # publish the name lexicon on risten.no (__Sjur__) # add missing parallel names for placenames (__linguists__) # add informative links between first names like Niillas and Nils (__linguists__) !!!9. Spellers !!OOo speller(s) TODO after the MS Office Beta is delivered: * add Hunspell data generation to the lexc2xspell (__Tomi__ - after the PLX data generation is finished) * study the Hunspell formalism in detail (__Børre, Sjur, Tomi__) !!Testing !Spelling Error Markup __TODO:__ * Manually mark test texts for typos (making them into gold standards) (__Steinar__) * change {{ccat}} to handle error/correction markup (__Tomi__) ** extract the whole text, both the original text and their corrections, in a tab separated file: {{erorrcorrection}} - just as typos.txt. Used for another type of speller testing. {{correction}} should be an emtpy string if there is no correction. *** done, see --help for documenation *** if the option doesn't show up in --help, the installed binary isn't updated. Please ask __Saara__ to compile and install if so. __Tomi__ should check whether the newest {{ccat}} is available in __victorio/G5__ * Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) ** add the wanted xml elements to the XSL header (__Saara__) (source info is the same as for regular docs): *** outworned (ie not suitable for speller testing any more, only for regression testing) *** lexicalised (all unknown, correctly spelled words added to lexicon) * Conduct tests on new beta versions on the basis of the unspoiled gold standard documents (__whoever has time__), and fill in data from testing (the testers: __who?__) ** __Steinar__ has done this sporadically * alternatively: make test scripts that will run the tests automatically, collect the numbers, and transform them into test results (__Sjur__) ** done, except that they don't work with our tailored Forrest * include the files already publically tested into the {{prooftest}} cataloge (__Steinar, Trond__) * test each version before beta release ** we don't have a public beta release candidate yet:( !Testing tools __TODO:__ * document the AppleScript testing tool (__Sjur__) * improve speller test bench (__Sjur__) ** first usable version done ** Forrest bug needs to be debugged (__Børre, Sjur__) ** integrate the ccat speller testing options in the Makefile (__Sjur, Tomi__) !Regression tests __TODO:__ * add extraction of all known spelling errors in the corpus (not the {{prooftest}} corpus), and check that they are properly corrected (__Børre, Sjur__) ** {{ccat}} now ready, it should be integrated in the Makefile (__Sjur, Tomi__) * test the {{typos.txt}} list, and check that all entries are properly corrected (__Børre, Sjur__) * consider how to do a regression __self-test__, ie, how to test the full wordlist (__Børre, Sjur__) ** extract all the base forms in the lexicon, and run them through the speller ** extract all SUB-marked entries, and run them through the lexicon *** integrate these in the make file (__Sjur__) !!Localisation We need to translate the info added to our front page (and a separate page) regarding the beta release. Also the press release needs to be translated. TODO: * translate beta release docs to {{sme}} (__Thomas__) * translate beta release docs to {{smj}} (__Thomas__) !!Lexicon conversion to the PLX format Postverbal clitics - the discussion moved in another direction, and the topic was forgotten. The status of the latest speller is: postverbal clitics __are__ recognised. Topic closed. !Numbers __TODO:__ * add numbers as figures to the PLX sources (__Børre__) ** done !Compounding restrictions How to include compounding restriction comment tags in the transducers: {{{ giv0ri:giv'ri ALBMI ; !+SgNomCmp +SgGenCmp +PlGenCmp => (using a perl script or similar) +SgNomCmp+SgGenCmp+PlGenCmpgiv0ri:giv'ri ALBMI ; ! }}} __TODO:__ # improve prefix conversion to PLX (__Tomi__) # improve middle noun conversion to PLX (__Tomi__) # improve noun + adjective PLX conversion: (__Tomi__) ## compounding stems - how do we generate them? Using the java client? {{+SgNomCmp+Cmpnd}} = {{sáme–}}, should give the correct compounding stem, shouldn't it? We want to __optionally__ go from: {{sáme- NLI}} to {{sáme NL}}: {{- NLI (->) NL}}, which means we should be able to extract correct compounding stems using xfst methods only. ## compounding tags - we need to obey them when making the transducers. Suggestion - see above. # add propernouns to xfst-based conversion ## done # make conversion test sample; add conversion testing to the make file (__Tomi__) ## to regression test / QA the PLX conversion. # improve number conversion (__Børre, Tomi__) # run xfst-based PLX conversion on victorio, make the result available on our public server (__Saara, Sjur__) ## done, but the web server (the humfak server) disk is too small - the present disk size is 10 Gb, we need at least 50 Gb. 100 Gb should give us room to grow. __Trond, Børre__ to ask for larger disk. !!Public Beta release We still have problems with sorting the PLX data. We will have Polderland sort it, and make the beta. We need to investigate where the error is in our tool chain. letter ''a'' from the sorted file: {{{ 14583071 al-le!ruo-vá-tjij-da-sis-ka NIR 14583072 al-le VI 14583073 al-le NIALR 14583074 al-le^Aruo-vá-tjijt NIR }}} __TODO:__ * improved {{smj}} speller (incl. derivations and compounds) (__Sjur, Tomi__) ** done in principle, see sorting problem above. * add numbers, compound restrictions to both spellers if time permits (__Børre, Tomi__) * add names to {{smj}} speller (__Sjur__) ** done * finish press release (__Sjur__) * add info to front page (incl. download links) (__Børre__) ** download page made, only needs to add the speller beta when it is ready. * write separate page with detailed info (incl. download links) (__Børre__) ** a separate page for the beta speller, with installation instructions, etc. * translate press release, web pages (__Børre, Thomas, whoever__) * collect a list of PR recipients, forward to Berit Karen Paulsen (__Børre, Sjur, Trond__) * test speller installers on Windows and Mac (__Børre__) ** we have a new multilingual Windows installer, which needs to be tested * update installer packages with latest speller lexicon (__Børre, Sjur__) ** we need to test the procedure, and make sure it works !!Version identification of speller lexicons __TODO:__ * make the date stamp reflect the compilation date automatically (__Sjur__) ** done !!!10. Other !!Corpus contracts TODO: * publish corpus contracts and project infra as open-source on NoDaLi-sta (__Sjur__) ** __delayed__ until the public beta is out the door !!Bug fixing __51__ open Divvun/Disamb bugs, and __23__ risten.no bugs !!New team member We had problems getting iChat to work from Per-Eric's computer. Needs to be resolved. 1. backup system: Skype for all -> Steinar. Second backup system: POT:) __TODO:__ * set up computer (__Børre, Sjur__) ** done * install all required software (__Børre, Sjur__) ** done * set up all user accounts (__Sjur, Trond__) ** done * fix iChat setup on Per-Eric's computer (__Sjur, Børre__) !!!11. Next meeting, closing The next meeting is 30.4.2007, 09:30 Norwegian time. The meeting was closed at 11:34. !!!Appendix - task lists for the next week !! Boerre [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Boerre.ics] * add {{sma}} texts to the corpus repository * collect a list of PR recipients, forward to Berit Karen Paulsen * run all known spelling errors in the corpus through the speller * add extraction of all known spelling errors in the regular corpus (not the {{prooftest}} corpus), and check that they are properly corrected * test speller installers on Windows and Mac ** we have a new multilingual Windows installer, which needs to be tested * update installer packages with latest speller lexicon * add numbers, compound restrictions to both spellers if time permits * add numbers as figures to the PLX sources * update and fix our documentation and infrastructure as __Steinar__ finds problem areas - low priority * find missing {{nob}} parallel texts in corpus * Forrest dispatcher bug with spelltest results needs to be debugged * study the Hunspell formalism in detail together with __Sjur__ and __Tomi__ * add extraction of all known spelling errors in the corpus (not the {{prooftest}} corpus), and check that they are properly corrected * test the {{typos.txt}} list, and check that all entries are properly corrected * Ask for larger disk for the webserver. * add info to front page (incl. download links) * write separate page with detailed info (incl. download links) (__Børre__) ** a separate page for the beta speller, with installation instructions, etc. * translate press release, web pages (__Børre, Thomas, whoever__) * update installer packages with latest speller lexicon (__Børre, Sjur__) * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Maaren [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Maaren.ics] * lexicalise actio compounds * Manually mark speller test documents for typos !! Per-Eric [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Per-Eric.ics] * work on {{smj}} !! Saara [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Saara.ics] * prepare more files for manual alignment * improve cgi-bin scripts ** add new features to the paradigm generator * add new XSL/XML headers for proofing test docs * compilation of verb lists * read the manual for graphical corpus interface and try to add files with Lars. * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Sjur.ics] * finish press release for the beta * collect a list of PR recipients * run all known spelling errors in the corpus through the speller * document the AppleScript testing tool * integrate regression self tests with the make file * make improved {{smj}} speller (incl. derivations and compounds) * improve speller test bench * update installer packages with latest speller lexicon * Forrest dispatcher bug with spelltest results needs to be debugged * integrate the ccat speller testing options in the make file * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Steinar [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Steinar.ics] * Beta testing: Align manually (shorter texts) * Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue * include the files already publically tested into the {{prooftest}} cataloge * Complete the semantic sets in sme-dis.rle * missing lists * Look at the actio compound issue when adding from missing lists * Align corpus manually * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Thomas.ics] * work with compounding * Lack of lowering before hyphen: Twol rewrite. * translate beta release docs to {{sme}} and {{smj}} * Add potential speller test texts * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Tomi [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Tomi.ics] * make improved {{smj}} speller (incl. derivations and compounds) * add numbers, compound restrictions to both spellers if time permits * make PLX conversion test sample; add conversion testing to the make file * improve number PLX conversion * improve prefix and middle-noun PLX conversion * check whether the newest {{ccat}} is available in __victorio/G5__ * integrate the {{ccat}} speller testing options in the Makefile * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond [iCal|/doc/admin/weekly/2007/Tasks_2007-04-23_Trond.ics] * Test the beta versions * Work on the parallel corpus issues ** fix {{sme}} texts in corpus this month ** find missing {{nob}} parallel texts in corpus, go through Saara's list * Postpone these tasks to after the beta: ** update the {{smj}} proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt ** Go through the Num bugs * collect a list of PR recipients * Ask for larger disk for the webserver. * [fix bugs!|http://giellatekno.uit.no/bugzilla].