!!!Meeting setup * Date: 29.05.2006 * Time: 09.30 Norw. time * Place: Where we are * Tools: SubEthaEdit, iChat !!!Agenda # Opening, agenda review # Reviewing the task list from two weeks ago # Documentation - divvun.no # Corpus gathering # Corpus infrastructure # Infrastructure # Linguistics # name lexicon infrastructure # Spellers # Other issues # Summary, task lists # Closing !!!1. Opening, agenda review, participants Opened at 09:37. Present: __Børre, Saara, Sjur, Thomas, Trond, Tomi__ Absent: __Maaren__ __Main secretary__: the whole concept dropped for now - working collaboratively makes it obsolete. Agenda accepted as is. !!!2. Updated task status since last meeting !! Børre * corpus collection: ** send out contracts with accompanying letter ** Gather public texts, preferrably also parallel ones *** Began cron job on http://galdu.org/samegillii ** Continue converting text from input format to our xml *** Done by convert2xml.pl, updated some .xsl files by hand ** convert nob and nno bible texts to be used as part of a parallel corpus *** Not done ** review the paratext2xml converter *** Not done ** convert smj NT to paratext *** Not done ** Send out letters to the rest of the Iđut authors *** Contact with Kurt Tore Andersen, writer of schoolbooks. Was positive. ** call Brita Kåven again towards the end of the week *** Not done ** contact Ája (Kåfjord) *** Not done ** create weekly cron job to mirror Odin URL and detect new/updated pages *** Not done ** Check the status & license of the corpus texts *** Done ** wait for R. Valkeapää, call him next week * public tender: ** assist with letter to the project board *** Read through the letter, participated in discussions * corpus access: ** zip an xml-stripped version of our free texts for to Olavi *** Done! ** meeting 23.5 at 9.30: discuss and decide upon the exact access policy we want *** Not done ** set up user account/corpus access for Olavi *** Not needed? ** set upp the unix group structure for corpus users (also __Saara, Trond__) *** To be done after the above-mentioned meeting * install latest SEE ** Done * install Gobby for __Thomas and Maaren__ ** Not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Maaren * On sick leave !! Saara * Create a parallel corpora of the new testaments. * add more texts to the graphical corpus interface * grammatical searchability in the graphical corpus interface ** these are not done * Implement links to parallel files in corpus header. * Implement parallel corpus upload in web upload script ** not done * Check the status & license of the corpus texts and change the default of the availability to "license" instead of "free" ** done * rerun the corpus conversion ** done on daily basis * Install Gobby ** can I have the tarball discussed in the last meeting? * update the corpus script(s) to only copy texts to the free/ dir which are explicitly marked as free ** done, some testing required (and cleaning the free-directory) * Add some language recognition flags to write into the xsl file ** not yet ready * rename corpus dirs, and create symlinks ** done, there is an unused directory gt which should be removed. * set upp the unix group structure for corpus users (also __Børre, Trond__) ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * public tender: ** write letter to the project board *** done * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer ** not yet done * name lexicon: ** implement editing functions ** finalise refactoring for multiple collections (regular search interface) *** nothing this week * change corpus-summary processing to generate smaller pages ** nothing yet - needs to be discussed * send bug report to Apple re filename matching and accented characters in Terminal ** done * meeting 23.5 at 9.30: discuss and decide upon the exact access policy we want ** delayed * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas * correct hyphenation of exceptions (sme) ** still working with this * work on compounding and derivation ** planned continues work on work-shop with Trond and Maaren * sme G3 issue ** nothing done * set up a linguistic workshop while Maaren is in Tromsø ** done !! Tomi * new proper name lexicon ** implement data synchronisation of proper nouns between risten.no and CVS *** not done ** XQuery refactoring and code development for our proper noun editor *** not done ** new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry *** not done * read aligner docu, install, provide feedback ** not done * install and test Gobby ** not done * Set up the mechanism for the hash-mark transducer package ** not done * add ccat option to analyse text while keeping the xml tags and structure ** not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond * better smj NT text * get fin, swe, nob and nno NT and OT in __paratext__ format ** Not done. * install aligner, test it and give feedback ** Discussed issue with Lars, will continue with __Saara__ * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer ** Not done * get/upgrade keys for __Børre's__ room for __Tomi__ and __Thomas__ Ordered, still not received answer. * Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org) ** No news. * Check the status & license of the corpus texts ** Done. * Work on the graphical corpus tag list ** Made progress. * send __Saara__ smj files for language recognition ** Talked to saara * create a short smj word list to help the trigram heuristics * Put __Saara__ and __Tero__ in contact with each other ** Not yet. * ask __Lars Nygård__ and __Tero Avellan__ to install Gobby 0.3 * set up a linguistic workshop while Maaren is in Tromsø ** done * meeting 23.5 at 9.30: discuss and decide upon the exact access policy we want ** delayed * set upp the unix group structure for corpus users (also __Børre, Saara__) ** delayed until the principles are settled * [fix bugs!|http://giellatekno.uit.no/bugzilla]. !!!3. Documentation TODO: * documentation on how to apply for a user account for the corpus repo (__Børre__) ** we will administer the corpus user accounts ourselves ** We first have to discuss and decide what we want before __Børre__ can write documentation (see further down) *** Not done. We didn't have the meeting. !!!4. Corpus gathering !!Collecting See a [previous meeting memo|Meeting_2006-01-16] for what's to be done. TODO: * Send out the rest of the letters (__Børre__) ** Working on this. Summing up the most important writers from Davvi Girji. New contracts: * none last 2 weeks !!Olavi Korhonen's Lule Sámi dictionary. Phoned Korhonen. He was willing to sign the contracts, and wanted some kind of access to our corpus. He wants to collect words for his Northern Sámi dictionary. TODO: * set up user account/corpus access for Olavi (__Børre__) ** we need the infrastructure for corpus access in place first * done: sent a Word document containing all free texts (2400 pages), as well as a list of all words (baseform) !!KIO Grafisk and the Iđut books __TODO__: * send letters to the authors (__Børre__) ** talked to Kurt Tore Andersen, who has written a lot of smj school books. He is interested in cooperation with us. !!Bible texts We will get text from Finland, but still haven't received any. Swedish html has arrived, no paratext. Norsk bibelselskap has not sent corrected New Testament versions for sme, and not paratext for nno/nob. TODO: * convert smj NT to paratext (__Børre__) * get fin, swe, nob and nno NT and OT in __paratext__ format. (__Trond__) !!Davvi Girji Called her last week. She said Davvi Girji os would give us permission to use texts by their authors. She wasn't sure if we could get the texts directly from Davvi Girji, because of copyrights of pictures and other artwork in the books. Said they (Davvi Girji workers) were going to have a meeting right after our conversation by phone, and take some kind of desicion on this case. Haven't heard anything from Kåven, though. __TODO__: * call Brita Kåven again towards the end of the week (__Børre__) * call the authors (__Børre__) !!Min Áigi The Min Áigi format should be dealt with: \@ingress etc should be dealt with for the .txt, but business as usal for the .doc files. __Saara__ has done the xsl conversion routine for the typographic tags. It still needs some fine tuning, as there are some @ tags that were not included in our initial list. __TODO__: * send bug report to Apple (typing filenames in Terminal does not match, moving filenames across plattforms via tarballs creates problems - the names are not the same!) (__Sjur__) ** not done yet * will have to complete metainformation !!Kåfjord Promised to send us texts. Some texts have arrived, but nothing from Ája. __TODO__: * contact Ája (__Børre__) ** not done !!Sámi Instituhtta __Børre__ contacted __Richard Valkeapää,__ the IT-consult at NSI. He put it on his todo list, as he would have to contact the person who has worked with the newspaper texts anyway. He said this would be done in the near future (within a month). TODO: * wait for R. Valkeapää, call him this week (__Børre__) !!!5. Corpus infrastructure !!User accounts and access We will probably have different kinds of users, some will only need access through the webinterface to the corpus, others might want and need access to the commandline, to utilize the corpus in an efficient way. This calls for an access policy for these users. TODO: * before anything else is done: Zip an xml-stripped version of our free texts and send them to Olavi. In that way, he gets something to work on, and we will get feedback (__Børre__) ** done * discuss and decide upon the exact access policy we want to give corpus users; meeting on Tuesday May 23, at 9.30 (__Børre, Sjur, Trond__) ** not done, new date: Thursday 1.6.2006, at 9.30 * set upp the unix group structure to open for a new category of users: Linguists with access to the texts on the servers. Some of __Børre, Saara, Trond__. * make a text-only corpus in the Oslo interface (dump the texts on omilia), thereby making them accessible to simple lexical lookup. The cost of doing this should be evaluated against what it takes waiting for the analysed corpus to be in place. (__Saara, Trond__) ** Discussion on this issue after this meeting !!Name change again? TODO: * rename corpus dirs, and create symlinks (__Saara__) ** done, remove the gt dir? Yes. !!Free and non-free texts More info in a [previous meeting memo.|/doc/admin/weekly/2006/Meeting_2006-03-13.html] TODO: * Check the status of the texts, again. (__Børre, Trond, Saara__) ** Complete (only new files need to be checked from now on) * Rerun the conversion afterwards (__Saara__ is the one with the magic spell) ** done automatically every night * update the script(s) to only copy texts to the free/ dir which are explicitly marked as free (__Saara__) ** done, but buggy !!More texts to the graphical corpus interface: We need to get the infrastructure complete to be able to do this, then it should be a piece of cake. TODO: * We would like to have more than the NT in [the graphical interface|http://omilia.uio.no/CE/sami/] (__Saara__) ** We add the largest texts first. * We would like to have grammatical searchability, not only POS. (__Saara__, __Trond__ This presupposes a discussion with Oslo. (__Trond__ and __Saara__ to continue this discussion) * For Lule Sámi: We would like to have a parallel corpus interface with NT (text only). This presupposes better quality texts (__Trond, Børre__) ** Better Lule NT text still not made. * The list of good candicates: The longest (admin) texts. ** We need a new option in ccat for analysing text while still keeping the xml tags and structure (__Tomi__) ** xml texts number

, preprocess finds <.> and ccat numbers them as . ** Then the aligner aligns... Top-three priorities: # Finish the tag unification (korpustags.txt) (__Trond__) # change ccat to be able to create the right input for the corpus analysis (xml- tagged output) (__Tomi__). Work estimate: a few (2-3?) days - nothing this week, we will re-evaluate (and schedule) next Monday) # add text to the server (__Lars__) !!Language recognition TODO: * refine language recognition (__Saara__) ** in progress, continue discussion in [249|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=249] * create a short smj word list to help the trigram heuristics (__Trond__) * send __Saara__ smj files (__Trond__) ** Call Árran again (__Trond__, then handle it over to Sámediggi, that is, to __Børre__) * Add some flag to write into the xsl file (__Saara__) ** unfinished Some Lule Sami text is found on [Infonuorras|http://www.infonuorra.no/index.php?lang=ls] site. !!Corpus summary TODO: * trim generated pages (now sme generates a table with 10 000 entries!) ** suggestion: lump together files with content less than X paragraphs (X < 5?) !!Proofed vs unproofed corpus files The Min Áigi material contains partially parallel unproofed vs proofed documents. We need to find a decent way of handling this parallellism, and preferably a way to make use of the information contained in the diff between the two versions. At best, we should be able to automatically extract the corrections made, generating an XML document that contains correction markup as discussed in the newsthread "Corpus DTD: corrections" (from 17.03.2006). I don't know how much work this is, or whether it is at all possible, but let's discuss it. For now there are so few such files that it hardly pays off, but __Saara__ will make a list of these files to evaluate the potential benefit. TODO: * count the number of parallel files of type unproofed/proofread (__Saara__) !!Aligner __Trond__ and __Saara__ will continue this issue. !!!6. Infrastructure !!Paradigm generation Goal: Reuse Greenlandic code for paradigm generation. TODO: * Put __Saara__ and __Tero__ in contact with each other (__Trond__) ** still open * The paradigm generator should also have an xml-out option (for use in e.g. electronic dictionaries) - on hold till we know more about the generator !!Hyphenator __Thomas__ is finished with adding ^ tags to the __smj__ noun file, and has continued working on the __sme__ noun file. __Trond__ and __Thomas__ have been working on the smj rule component, and have improved both the treatment of weak grade consonant clusters (preconsonantal geminates) and on some loan word patterns. TODO: * correct the treatment of hyphenation of word boundaries and exceptions (fst gymnastics) (__Sjur, Trond__) ** Still not done. * Update the sme and sma rule sets with the insights gained from smj updates. !!!7. Linguistics !!General issues Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org, or the Lyndi England issue, Aftenposten Obj Org WoA Pub) Possible rule: {{{ If Plc then Obj }}} !!North Sámi TODO: * set up a linguistic workshop while Maaren is in Tromsø (and remember that she is on sick leave all the time) (__Trond, Thomas, Maaren__) ** done, more needed: * Maaren is in Tromsø this week, we should continue our linguistic discussions. !!Lule Sámi There are some open issues in the marginal area of the smj transducer: * numerals, e.g. Our poor treatment of number words becomes more visible as we get real texts (guhttatuvsánanielljatjuodegålmmålåkvihtta is not recognised, etc.). * names => waiting for the new name lexicon * compounds? Shortening here as well, but not in written language (some lexicalised exceptions, as in sme) => nothing to discuss/implement * loanwords? We should consider importing the ^LOAN words from sme and Lule-ify them. Otherwise, loan words should come from corpora. TODO: * 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks * add proper numeral analysis/treatment * add loanwords (e.g. latin -ere verbs) !!!8. Name lexicon infrastructure TODO: # finish refactoring for multiple collections in the search interfarce (__Sjur__) ## nothing done last week # develop the needed XQueries and interface (__Sjur, Tomi__) ## progressing, done some, haven't commited (adding new term, create-termc-entry.xq) # data synchronisation between risten.no and the cvs repo (__Tomi__) ## discussion started on eXist-list, we'll wait a couple of days to see what's coming out of it, and if nothing useful to us, we'll add our use case with questions # test and review when ready Timeline: * One-time conversion lexc2xml (sme to common) ** done (must be redone at D-day, but the *script* is done) * editing functions in risten.no - what about editing in emacs/see/other editors ** in the works * Automatic converson: xml2lexc (modulo language) (based on ccat) (ready to be executed) (__Tomi__) ** not done !!!9. Spellers We will remove this speller section till we have something to report. !!!10. Public tender Finnut called, and here's their evaluation: if we think that the offers are incomplete or otherwise not fully acceptable, we can enter negotiations with the companies, effectively cancelling the current public tender. This can __only__ be done as long as we don't change the public tender document (that is, the foundation for the public tender) - if it is changed, we have to announce the whole competition again, with the usual 53 days minimum deadline for applications. __Sjur__ will send an e-mail to the project board, outlining the different aspects of the two offers, and ask about their opinion on the following questions: * accept the offers as is, or enter negotiations? * if accept, which one? TODO: * write e-mail to the project board, ask for their opinion regarding the offers (__Sjur__) ** done, waiting for answers from the board !!!11. Other !!Summer vacation || Who || When | Børre | August | Linda | ? | Maaren | ? | Saara | July | Sjur | at least some in July, but still open | Thomas | 3.7 - 7.8 | Trond | 3.7 - 14.8 (last two weeks off at summer school) | Tomi | 8.7 - 16.7, more? !!Bug fixing __43__ open Divvun/Disamb bugs (two down!), and __25__ risten.no bugs Guess: 1/3 of the bugs are fixed already (?) Please help __Saara__ with [bug 279|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=279] (Perl locale). Not much help... __Saara__ will contact __Roy__ on this issue. After the corpus issues have been somewhat settled, we should do a bug barnraising. ... and then a new one after the name lexicon is fixed. !!Gobby 0.3 is working fine on Mac, Linux and Windows. Should be installed on all computers c.f. [http://darcs.0x539.de/trac/obby/cgi-bin/trac.cgi/wiki/InstallationGuide] (our preinstalled Xcode veriosn is 2.0, must be 2.1): * Børre - ok by copying {{/opt/local/}} from __Trond__ * Maaren - Børre to do it * Saara - todo * Sjur - ok * Thomas - Børre to do it * Tomi - not working * Trond - ok Easy way out when the standard Darwin Ports installation isn't working: just get a copy of {{/opt/local/}} from __Børre__. __Trond__ should ask __Lars Nygård__ and __Tero Avellan__ to install Gobby as well; has asked __Per Langgård__. !!SEE 2.5 extensions * syntax coloring of meeting memos * script to extract tasks? TODO: * give the jspwiki syntax colouring mode to all (__Sjur__) !!!12. Next meeting, closing 06.06.2006 09:30 Closed at 11:20 !!!Appendix - task lists for the next week !! Børre * corpus collection: ** send out contracts with accompanying letter ** Gather public texts, preferrably also parallel ones ** Continue converting text from input format to our xml ** convert nob and nno bible texts to be used as part of a parallel corpus ** review the paratext2xml converter ** convert smj NT to paratext ** Send out letters to the rest of the Iđut authors ** call Brita Kåven again towards the end of the week ** contact Ája (Kåfjord) ** create weekly cron job to mirror Odin URL and detect new/updated pages ** wait for R. Valkeapää, call him this week * public tender: * corpus access: ** meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want ** set upp the unix group structure for corpus users (also __Saara, Trond__) * install Gobby for __Thomas and Maaren__, send {{/opt/local/}} tarball to __Saara__ * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Maaren * On sick leave !! Saara * Create a parallel corpora of the new testaments. * add more texts to the graphical corpus interface * grammatical searchability in the graphical corpus interface * Implement links to parallel files in corpus header. * Implement parallel corpus upload in web upload script * Install Gobby * clean the {{free/}} dir and test copying of only free texts to it * Add some language recognition flags to write into the xsl file * set upp the unix group structure for corpus users (also __Børre, Trond__) * count and examine pairs of proofread and nonproofed MinAigi files * evaluate addition of a text-only corpus in the Oslo interface (with __Trond__) * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * public tender: ** waiting for answers from the board, then act * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer * name lexicon: ** implement editing functions ** finalise refactoring for multiple collections (regular search interface) * change corpus-summary processing to generate smaller pages * send bug report to Apple re filename matching and accented characters in Terminal * meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas * correct hyphenation of exceptions (sme) * work on compounding and derivation * sme G3 issue !! Tomi * new proper name lexicon ** data synchronisation of proper nouns between risten.no and CVS ** XQuery refactoring and code development for our proper noun editor ** new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry * read aligner docu, install, provide feedback * install and test Gobby * Set up the mechanism for the hash-mark transducer package * add ccat option to analyse text while keeping the xml tags and structure * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond * better smj NT text * get fin, swe, nob and nno NT and OT in __paratext__ format * install aligner, test it and give feedback * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer * get/upgrade keys for __Børre's__ room for __Tomi__ and __Thomas__ * Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org) * Work on the graphical corpus tag list * send __Saara__ smj files for language recognition * create a short smj word list to help the trigram heuristics * Call Árran again, and then transfer smj corpus discussions to __Børre__ * Put __Saara__ and __Tero__ in contact with each other * ask __Lars Nygård__ and __Tero Avellan__ to install Gobby 0.3 * continue linguistic discussions while Maaren is in Tromsø * meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want * set upp the unix group structure for corpus users (also __Børre, Saara__) * evaluate addition of a text-only corpus in the Oslo interface (with __Saara__) * [fix bugs!|http://giellatekno.uit.no/bugzilla].