!!!Meeting setup * Date: 14.05.2007 * Time: 09.30 Norw. time * Place: Internet * Tools: SubEthaEdit, iChat !!!Agenda # Opening, agenda review # Reviewing the task list from last week # Documentation - divvun.no # Corpus gathering # Corpus infrastructure # Infrastructure # Linguistics # name lexicon infrastructure # Spellers # Other issues # Summary, task lists # Closing !!!1. Opening, agenda review, participants Opened at 12:50. Present: __Børre, Maaren, Per-Eric, Sjur, Steinar, Thomas, Tomi, Trond__ Absent: __Saara__ Agenda accepted as is. !!!2. Updated task status since last meeting !! Børre * add {{sma}} texts to the corpus repository ** not done * collect a list of PR recipients, forward to Berit Karen Paulsen ** not done * run all known spelling errors in the prooftest corpus through the speller ** not done, on new speller last week either * add extraction of all known spelling errors in the regular corpus (not the {{prooftest}} corpus), and check that they are properly corrected ** not done * update installer packages with latest speller lexicon ** not done * add numbers, compound restrictions to both spellers if time permits * update and fix our documentation and infrastructure as __Steinar__ finds problem areas - low priority ** not done * find missing {{nob}} parallel texts in corpus ** not done * study the Hunspell formalism in detail together with __Sjur__ and __Tomi__ ** nothing new * add info to front page (incl. download links) ** not done * write separate page with detailed info (incl. download links) (__Børre__) ** not done ** a separate page for the beta speller, with installation instructions, etc. ** not done * translate press release, web pages (__Børre, Thomas, whoever__) ** not done * ask MacOffice for larger disks for the G5 ** done * follow up the server OS on victorio ** __Roy Dragseth__ said there won't be time to do anything untill June. * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** not done * other ** set up __Thomas__ computer for __Inga Mikkelsen__ ** The G5 corrupted data when compiling the spellers. Searched for the fault and found faulty RAM on the G5 ** compiled spellers last week. Produced a couple of spellers with flaws. Building new verb- and propernoun sme list now. !! Maaren * lexicalise actio compounds **not done * Manually mark speller test documents for typos **working !! Per-Eric * expand the smj typos list **continued working * add missing smj words **continued working * research the ''–k'' ending, whether it is a clitic or a regular inflection ** done some research by the telephone with __Karin Tuolja__ and __Nils Olof Sortelius__ about it, example ''buorrek'', ''buorrege'' and ''buorrega'', they mean these example words are clitics and so do i mean too !! Saara * improve cgi-bin scripts ** add new features to the paradigm generator ** paradigm generator for Lule Sámi *** in progress * add new XSL/XML headers for proofing test docs ** not done * compilation of verb lists ** not done * read the manual for graphical corpus interface and try to add files with Lars. ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * finish press release for the beta ** still unfinished * collect a list of PR recipients ** still open * run all known spelling errors in the corpus through the speller ** still open * document the AppleScript testing tool ** still open * integrate regression self tests with the make file ** still open * make improved {{smj}} speller ** finally done (__Børre__ compiled it - more details below) * improve speller test bench ** still open * update installer packages with latest speller lexicon ** made improvements, both Win and Mac installer packages now in CVS. * integrate the ccat speller testing options in the make file ** still open * fix internet setup for __Per-Eric's__ satelite modem ** found some more info, sent it to the modem maker, waiting for feedback * ask for Linux version of the Polderland command-line speller (for victorio) ** did it. It will cost money, do we need it? * ask for mklex for Linux (victorio) from Polderland (for victorio) ** did it. It will cost money, do we need it? * look over the Bugzilla status mails ** waiting for __Børre__ * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Steinar * Beta testing: Align manually (shorter texts) * Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue ** continued working * include the files already publically tested into the {{prooftest}} cataloge ** not done, the files are deleted * Complete the semantic sets in sme-dis.rle ** no work this week * missing lists ** work this week * Look at the actio compound issue when adding from missing lists ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas * work with compounding ** worked and still working * Lack of lowering before hyphen: Twol rewrite. ** not done * translate beta release docs to {{sme}} and {{smj}} ** not done * Add potential speller test texts ** not done * research the ''–k'' ending, whether it is a clitic or a regular inflection ** done some research * [fix bugs!|http://giellatekno.uit.no/bugzilla] ** worked some !! Tomi * make improved {{smj}} speller (incl. derivations and compounds) ** improved plx conversion * add numbers, compound restrictions to both spellers if time permits ** improved number conversion * make PLX conversion test sample; add conversion testing to the make file ** not done * improve number PLX conversion ** see above * improve prefix and middle-noun PLX conversion ** not done * integrate the {{ccat}} speller testing options in the Makefile ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] * other ** renamed plx filters ** improved propernoun conversion !! Trond * Work on the parallel corpus issues ** Not this week. * Add Steinars corr texts to the relevant catalogue ** No texts added. * Postpone these tasks to after the beta: ** update the {{smj}} proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt ** Go through the Num bugs * collect a list of PR recipients ** Not done. * check with __Lars__ when more aligned texts are ready * Follow up the server OS on victorio ** Done. * [fix bugs!|http://giellatekno.uit.no/bugzilla]. ** Worked on bug issues, yes. !!!3. Documentation Nothing new. The open documentation issues fall into these three categories: * Beta documentation for testers * Documentation for the online corpora * General documentation improvement after Steinar's test (for open-source release) TODO: * write form to request corpus user account (__Børre, Sjur, Trond__) ** delayed till after the beta release * document how to apply for access to closed corpus, and details on the corpus and its use in general (__Børre, Sjur, Trond__) ** delayed till after the beta release * correct and improve it based on feedback from __Steinar__ (__Børre__) ** low priority * beta documentation (see separate beta section below) !!!4. Corpus gathering __Børre__ received three books from __Jovnna-Ánde Vest__, ''Árbbolaččat'' 2 & 3, ''Sipirjá''. __Aage Solbakk__ and __Elle-Márjá Vars__ promised to look through their computers and floppies and send us texts. __Per-Eric__ has been talking with __Olavi Korhonen__, which promised to sign the contract, and move the issue forward (regarding his dictionary). Has also talked to __Susanna Angeus-Kuoljok__ and __Karin Tuolja__ about the book ''gårnåtja''. __Davvi Girji__ would like us to contact them to bring the text licensing forward. __Børre__ and __Sjur__ will do that. TODO: * {{sme}} texts: no new additions, fix corpus errors during this month (__Børre, Trond, Saara__) * missing {{nob}} parallel texts should be added if such holes are found (__Børre, Trond__) * Go through the list of missing or errouneous {{nob}} texts, based upon __Saara's__ perfect list (__Børre, Trond__) * add {{sma}} texts to the corpus repository (__Børre__) * contact ''Davvi Girji / Mikal Aase'' (__Børre, Sjur__) !!!5. Corpus infrastructure !!Alignment __TODO__ * check with __Lars__ when more texts are ready (__Trond__) !!!6. Infrastructure __TODO:__ * update and fix our documentation and infrastructure as __Steinar__ finds problem areas (__Børre__) * fix internet setup for __Per-Eric's__ satelite modem (__Sjur, Børre__) ** this influences iChat, SEE sharing, and ARD connetions *** did some investigation last week, waiting for an answer from the modem maker * Add a Lule Sámi paradigm generators, and link the p-smj.en.html page to it. (__Saara__) * Translate the paradigm pages to Sámi, fix all the se links (__Børre, Trond__) ** we have separate documents, but they are not translated, and not linked !!!7. Linguistics !!North Sámi TODO: * lexicalise actio compounds. Example: ''vuolggasadji'' vs. ''vuolginsadji'' (__Maaren__) ** possibly turn on free compounding as part of the PLX conversions (ie free compounding in the spellers, but not in the analyzers/transducers) * fix stuorra-oslolaš lower case {{o}} (__Sjur, Thomas, Trond__) ** postponed till after the public beta !!Lule Sámi Clitics: # Suggest it or not? NOT. Issue closed. # Accept it or not? Issue open. Solution for now: Not accept it Solution for later, after investigating false positives: Accept it. Anyway: we need a proper description on when -k is allowed, and then make sure it only appears there (-k is an allomorph of -ge only appearing after XXX, where XXX is possibly a vowel, but we need to investigate XXX) Arguments AGAINST our solution: * Spiik ''Lulesamisk ordbok'': ** heller (med negering) ge; -k; -ga; -ge; -gen, (ex inte -) iv; i; ij etc. ga/ ge/ gen, * Spiik ''grammar'': ** ändelsen -k legges til ord som sluter på V, och ändelsen -ge legges till ord som slutar på konsonant. If Spiik is right (i.e., if what he says is valid for WRITTEN LANGAUGE, then we may go for b. This is the solution IF WE WANT -k/-ga. +Clt+ga:#ka ENDLEX ; twol rule changing ka to ga post cns and to k post vow TODO: * refine {{smj}} proper noun lexica, cf. the propernoun-smj-lex.txt (__Thomas, Trond__) * research the ''–k'' ending, whether it is a clitic or a regular inflection, and where it applies (__Per-Eric, Thomas__) !!!8. Name lexicon infrastructure Decisions made in Tromsø can be found in [this meeting memo.|/admin/physical_meetings/tromso-2006-08-propnoun.html] __TODO:__ # fix bugs in lexc2xml; add comments to the log element (__Saara__) # finish first version of the editing (__Sjur__) # test editing of the xml files. If ok, then: (__Sjur, Thomas, Trond__) # make terms-smX.xml <=== automatically from propernoun-sme-lex.xml (add nob as well) (the morphological section should be kept intact, in e.g. propernoun-sme-morph.txt) (__Sjur, Saara__) # convert propernoun-($lang)-lex.txt to a derived file from common xml files (__Sjur, Tomi, Saara__) # implement data synchronisation between [risten.no|http://www.risten.no] and the cvs repo, and possibly other servers (ie the G5 as an alternative server to the public risten.no - it might be faster and better suited than the official one; also local installations could be treated the same way) # start to use the xml file as source file # clean terms-sme.xml such that all names have the correct tag for their use (e.g. @type=secondary) (__Thomas, Maaren, linguists__) # merge placenames which are errouneously in different entries: e.g. Helsinki, Helsingfors, Helsset (__linguists__) # publish the name lexicon on risten.no (__Sjur__) # add missing parallel names for placenames (__linguists__) # add informative links between first names like Niillas and Nils (__linguists__) !!!9. Spellers !!OOo speller(s) TODO after the MS Office Beta is delivered: * add Hunspell data generation to the lexc2xspell (__Tomi__ - after the PLX data generation is finished) * study the Hunspell formalism in detail (__Børre, Sjur, Tomi__) !!Testing !Spelling Error Markup __TODO:__ * Manually mark test texts for typos (making them into gold standards) (__Steinar__) * Set up ways of adding meta-information (source info, used in testing or not, added to lexicon or not) * Conduct tests on new beta versions on the basis of the unspoiled gold standard documents (__whoever has time__), and fill in data from testing (the testers: __who?__) * include the files already publically tested into the {{prooftest}} cataloge (__Steinar, Trond__) * test each version before beta release !Testing tools __TODO:__ * document the AppleScript testing tool (__Sjur__) * improve speller test bench (__Sjur__) ** integrate the ccat speller testing options in the Makefile (__Sjur, Tomi__) * ask for Linux version of the Polderland command-line speller (for victorio) (__Sjur__) ** done, waiting for a price quote (it is outside the present contract) !Regression tests __TODO:__ * add extraction of all known spelling errors in the corpus (not the {{prooftest}} corpus), and check that they are properly corrected (__Børre, Sjur__) * test the {{typos.txt}} list, and check that all entries are properly corrected (__Børre, Sjur__) * consider how to do a regression __self-test__, ie, how to test the full wordlist (__Børre, Sjur__) ** extract all the base forms in the lexicon, and run them through the speller ** extract all SUB-marked entries, and run them through the lexicon *** integrate these in the make file (__Sjur__) !!Localisation We need to translate the info added to our front page (and a separate page) regarding the beta release. Also the press release needs to be translated. TODO: * translate beta release docs to {{sme}} (__Thomas__) * translate beta release docs to {{smj}} (__Thomas__) !!Lexicon conversion to the PLX format __Børre__ has been doing the whole PLX conversion process on his own computer under Linux, skipping the gzip stage. This should speed up the process quite a bit, but is problematic from a legal point of view. The proper solution is to get a new license from either Polerland or Xerox that will let us run the whole build process on one single computer, be it victorio or the G5. Børre has been working with sme files from late april, i.e., we need (and already got?) better routines for file input. __TODO:__ * ask MacOffice for larger disks for the G5 (__Børre__), and report about possible RAM-related problems, and the warranty on the RAM. ** done * follow up on larger disks for victorio (__Børre, Trond__) * follow up on newer server OS on victorio (__Børre, Trond__) ** probably not before June, hopefully then * ask for mklex for Linux (victorio) from Polderland (__Sjur__) ** waiting for a price quote. * ask Xerox for a commercial lisense for the xfst tools on the G5 (__Sjur__) !Compounding restrictions How to include compounding restriction comment tags in the transducers: {{{ giv0ri:giv'ri ALBMI ; !+SgNomCmp +SgGenCmp +PlGenCmp => (using a perl script or similar) +SgNomCmp+SgGenCmp+PlGenCmpgiv0ri:giv'ri ALBMI ; ! }}} __TODO:__ # improve prefix conversion to PLX (__Tomi__) # improve middle noun conversion to PLX (__Tomi__) # improve noun + adjective PLX conversion: (__Tomi__) ## compounding stems - how do we generate them? Using the java client? {{+SgNomCmp+Cmpnd}} = {{sáme–}}, should give the correct compounding stem, shouldn't it? We want to __optionally__ go from: {{sáme- NLI}} to {{sáme NL}}: {{- NLI (->) NL}}, which means we should be able to extract correct compounding stems using xfst methods only. ## compounding tags - we need to obey them when making the transducers. Suggestion - see above. # make conversion test sample; add conversion testing to the make file (__Tomi__) ## to regression test / QA the PLX conversion. # improve number conversion (__Børre, Tomi__) # ask for larger disk for the web server (__Trond, Børre__) !!Public Beta release __TODO:__ * working and updated {{smj}} speller (__Sjur, Tomi__) * add numbers, compound restrictions to both spellers if time permits (__Børre, Tomi__) * finish press release (__Sjur__) * add info to front page (incl. download links) (__Børre__) ** download page made, only needs to add the speller beta when it is ready. * write separate page with detailed info (incl. download links) (__Børre__) ** a separate page for the beta speller, with installation instructions, etc. * translate press release, web pages (__Børre, Thomas, whoever__) * collect a list of PR recipients, forward to Berit Karen Paulsen (__Børre, Sjur, Trond__) * test final beta speller installers on Windows and Mac (__Børre__) * update installer packages with latest speller lexicon (__Børre, Sjur__) ** we need to test the procedure, and make sure it works !!!10. Other !!Summer vacation When are we taking it? Please fill in the table below: || Name || Starting || Ending | Børre | x| x | Maaren | x| x | Per-Erik | x| x | Saara | x| x | Sjur | x| x | Steinar | x| x | Thomas | x| x | Tomi | 9.7 | 5.8 | Trond | x| x Divvun people also need to send the dates to __Julie Eira__. !!Corpus contracts TODO: * publish corpus contracts and project infra as open-source on NoDaLi-sta (__Sjur__) ** __delayed__ until the public beta is out the door !!Bug fixing __37__ open Divvun/Disamb bugs, and __23__ risten.no bugs __TODO:__ * look over the Bugzilla status mails (__Børre__) !!!11. Next meeting, closing The next meeting is 21.5.2007, 09:30 Norwegian time. The meeting was closed at 16:05. !!!Appendix - task lists for the next week !! Boerre * add {{sma}} texts to the corpus repository * collect a list of PR recipients, forward to Berit Karen Paulsen * run all known spelling errors in the prooftest corpus through the speller * add extraction of all known spelling errors in the regular corpus (not the {{prooftest}} corpus), and check that they are properly corrected * update installer packages with latest speller lexicon * update and fix our documentation and infrastructure as __Steinar__ finds problem areas - low priority * study the Hunspell formalism in detail * test the {{typos.txt}} list, and check that all entries are properly corrected * add info to front page (incl. download links) * write separate page with detailed info (incl. download links) ** a separate page for the beta speller, with installation instructions, etc. * translate press release, web pages * update installer packages with latest speller lexicon * follow up the server OS on victorio * contact ''Davvi Girji / Mikal Aase'' * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Maaren * lexicalise actio compounds * Manually mark speller test documents for typos !! Per-Eric * expand the smj typos list * add missing smj words * research the ''–k'' ending, whether it is a clitic or a regular inflection !! Saara * improve cgi-bin scripts ** add new features to the paradigm generator ** paradigm generator for Lule Sámi * add new XSL/XML headers for proofing test docs * compilation of verb lists * read the manual for graphical corpus interface and try to add files with Lars. * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * finish press release for the beta * collect a list of PR recipients * run all known spelling errors in the corpus through the speller * document the AppleScript testing tool * integrate regression self tests with the make file * make improved {{smj}} speller * improve speller test bench * update installer packages with latest speller lexicon * integrate the ccat speller testing options in the make file * fix internet setup for __Per-Eric's__ satelite modem * ask for Linux version of the Polderland command-line speller (for victorio) * ask for mklex for Linux (victorio) from Polderland (for victorio) * look over the Bugzilla status mails * contact ''Davvi Girji / Mikal Aase'' * ask Xerox for a commercial lisense for the xfst tools on the G5 * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Steinar * Beta testing: Align manually (shorter texts) * Manually mark speller test texts for typos (making them into gold standards), add the texts to the orig/catalogue * Complete the semantic sets in sme-dis.rle * missing lists * Look at the actio compound issue when adding from missing lists * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas * work with compounding * Lack of lowering before hyphen: Twol rewrite. * translate beta release docs to {{sme}} and {{smj}} * Add potential speller test texts * research the ''–k'' ending, whether it is a clitic or a regular inflection * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Tomi * add compound restrictions to both spellers * make PLX conversion test sample; add conversion testing to the make file * improve prefix and middle-noun PLX conversion * integrate the {{ccat}} speller testing options in the Makefile * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond * Work on the parallel corpus issues * Add Steinars corr texts to the relevant catalogue * Postpone these tasks to after the beta: ** update the {{smj}} proper noun lexicon, and refine the morphological analysis, cf. the propernoun-smj-lex.txt ** Go through the Num bugs * collect a list of PR recipients * check with __Lars__ when more aligned texts are ready * [fix bugs!|http://giellatekno.uit.no/bugzilla].