!!!Meeting setup * Date: 15.05.2006 * Time: 09.30 Norw. time * Place: Where we are * Tools: SubEthaEdit, iChat !!!Agenda # Opening, agenda review # Reviewing the task list from two weeks ago # Documentation - divvun.no # Corpus gathering # Corpus infrastructure # Infrastructure # Linguistics # name lexicon infrastructure # Spellers # Other issues # Summary, task lists # Closing !!!1. Opening, agenda review, participants Opened at 09:54. Present: __Børre, Saara, Sjur, Thomas, Trond, Tomi__ Absent: __Maaren__ Main secretary: __Tomi/Børre__ Agenda accepted as is. !!!2. Reviewing the task list from the last meeting !! Børre * send out contracts with accompanying letter ** Some done * Gather public texts, preferrably also parallel ones ** Not done * Continue converting text from input format to our xml ** Done * convert nob and nno bible texts to be used as part of a parallel corpus ** Not done * review the paratext2xml converter ** Not done * convert smj NT to paratext ** Not done * Send out letters to the Iđut authors ** Some done * write docu for how to apply for a corpus user account (forms, recipients, etc) * remove old corpus files from gt/sme/corp/ after __Trond__ has cleaned it * mirror Odin URL (create cron task to do it automatically?) ** Not done * read & evaluate received offers ** Done * telephone meeting Wednesday with Finnut ** What happened with this? Sjur had a short meeting alone, as we are not ready for the concluding meeting yet. * Check the status & license of the corpus texts ** Not done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Maaren * Working on missing list ** Done. !! Saara * Create a parallel corpora of the new testaments. * grammatical searchability in the graphical corpus interface * move to bugzilla: ** set up a weekly cron script to make free texts available ** add a possibility to upload whole documents for hyphenation (and also analysis, generation, etc) ** add a log of every word/text uploaded/hyphenated/analyzed etc. *** done * Implement links to parallel files in corpus header. ** not done * Implement parallel corpus upload in web upload script ** not done * Implement turning off the language recognition in the xsl-file (and corpus.dtd). ** not done * Refine language recognition ** not done * Investigate the decomposed Unicode characters in file names -problem. ** solved * Correct decomposed Unicode in Min Áigi file names ** done * Check the status & license of the corpus texts ** not finished yet, some discussion: bug 284 * Rerun corpus conversion ** done extensively * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * read & evaluate received offers ** done, we need more info from the tenderers - questions sent, with a deadline on May 17. * telephone meeting Wednesday with Finnut ** had a short meeting (without __Børre__), clarifying what's left regarding technical issues. Real meeting postponed till we have all the needed answers, and can complete the technical evaluation * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer ** nothing * name lexicon: ** implement editing functions *** started ** write down the most common editing scenarios, to guide development *** done, quite helpful !! Thomas * correct hyphenation of exceptions ** not done * correct hyphenation of smj -st- ** not done * work on compounding and derivation ** worked * smj G3 issue ** not done * sme G3 issue ** not done !! Tomi * move to Bugzilla: ** aspell UTF-8 suffix bug *** not done ** Document aspell infrastructure: finish doc/proof/spelling/X-spell/aspell.xml (it's almost done, but there are a couple of loose ends) *** not done * new proper name lexicon ** implement data synchronisation of proper nouns between risten.no and CVS *** not done ** XQuery refactoring and code development for our proper noun editor *** not done ** new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry *** not done ** write down the most common editing scenarios (to be used as guides for making the editing interface) (adding / changing ) *** done * read aligner docu, install, provide feedback ** not done * install and test Gobby, install new version of SEE ** not done * Set up the mechanism for the hash-mark transducer package ** not done * read & evaluate received offers ** done * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond * better smj NT text, get fin NT texts ** Not done. * get a key for __Maaren__ in May ** Well, done, sort of. * install aligner, test it and give feedback ** Not done. * correct hyphenation of word boundaries and exceptions with __Thomas__ ** Not done (but __Thomas__ has worked on this). * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer ** Not done. * get/upgrade keys for __Børre's__ room for __Tomi__ and __Thomas__ ** Not done. * [fix bugs!|http://giellatekno.uit.no/bugzilla]. ** Done some, but suggesting a collective effort wrt. the bugs. * Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org) ** Not done. * write down the most common editing scenarios (to be used as guides for making the editing interface) (adding / changing ) ** Done, wasn't it? * read & evaluate received offers ** Work in progress. * Check the status & license of the corpus texts ** Must check whether all is done. !!!3. Documentation TODO: * documentation on how to apply for a user account for the corpus repo (__Børre__) ** The item will be moved to Bugzilla !!!4. Corpus gathering !!Collecting See a [previous meeting memo|Meeting_2006-01-16] for what's to be done. TODO: Send out the rest of the letters (__Børre__) New contacts: * Elle Márjá Vars (sme, 38 titles in Bibsys, novels, plays, both iđut and Davvi Girji, one translated into smj and sma, several into nob) !!Odin * __Børre__ will set up a weekly cron job to mirror the URL mentioned in Sæths e-mail, and add missing files to the corpus. !!Olavi Korhonen's Lule Sámi dictionary. * No news this week TODO: __Børre__ to contact Olavi Korhonen and Kuhmunen !!KIO Grafisk and the Iđut books __TODO__: * send letters to the authors (__Børre__) ** one contacted, the rest this week !!Bible texts We will get text from Finland, but still haven't received any. We have got the Swedish text from Sweden. As for the last html versions from Norway, __Trond__ has not contacted them last week. Swedish html has arrived, no paratext. Norsk bibelselskap has not sent corrected New Testament versions for sme, and not paratext for nno/nob. TODO: * convert smj NT to paratext (__Børre__) * get fin, swe, nob and nno NT and OT in __paratext__ format. (__Trond__) !!Davvi Girji A talk with Brita Kåven, revealed that they would have a look at the contracts __after__ easter. Has been away, __Børre__ will call her again today. !!Min Áigi Agreement: they will send us updates each month. Standard license. We have problems with Unicode characters in filenames. This was solved once before, and we need to look at this again. The old Bugzilla issue should be reopened. The bug was reopened: http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=76 The Min Áigi format should be dealt with: \@ingress etc should be dealt with for the .txt, but business as usal for the .doc files. Done so far: Trond and Saara? Trond made a list of tags, and Saara will make the xsl conversion routine for the typographic tags. Saara is implementing the tag format. __TODO__: * reopen Bugzilla issue, and study the previous discussion and solution regarding Unicode characters in filenames (__Saara__) ** done * add filename extensions to files not having one ** done * Space in file names should be changed to underscore (and not to hyphen!). ** done * send bug report to Apple (typing filenames in Terminal does not match, moving filenames across plattforms via tarballs creates problems - the names are not the same!) (__Sjur__) * make xsl conversion routine for the typographic tags (__Saara__) !!Kåfjord Promised to send us texts. Some texts have arrived, but nothing from Ája. __TODO__: __Børre__ will contact them. !!Sámi Instituhtta Børre contacted Richard Valkeapää, the IT-consult at NSI. He put it on his todo list, as he would have to contact the person who has worked with the newspaper texts anyway. He said this would be done in the near future (within a month). !!!5. Corpus infrastructure http://www.divvun.no/doc/lang/corp/corpus-summary.html TODO: !!Changes and updates because of the Divvun public tender User account admin and infra: see [previous memo|/doc/admin/weekly/2006/Meeting_2006-03-06.html]. TODO: see above under ''Documentation.'' Automatic build of the content of our corpus repo: also see [previous memo|/doc/admin/weekly/2006/Meeting_2006-03-06.html]. TODO: * make free texts available ** done weekly by a cron script (but only if there are new files) (__Saara__) *** move to bugzilla: done * Make a link, easily available, to these texts. ** done as a downloadable tar package. Name change again? __ Trond__ has come up with some new suggestion: {{{ gt -> gtbound/ gtbound -> bound/ gtfree -> free/ }}} NB! keep symlinks from the old names to the new names for now. !!Free and non-free texts More info in a [previous meeting memo.|/doc/admin/weekly/2006/Meeting_2006-03-13.html] TODO: * Check the status of the texts, again. (__Børre, Trond, Saara__) * Rerun the conversion afterwards (__Saara__ is the one with the magic spell) * check against bugs [273|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=273] and [272|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=272] ** these are now fixed. * update the script(s) to only copy texts to the free/ dir which are explicitly marked as free (__Saara__) !!More texts to the graphical corpus interface: TODO: * We would like to have more than the NT in [the graphical interface|http://omilia.uio.no/CE/sami/] (__Saara__) ** We add the largest texts first. * We would like to have grammatical searchability, not only POS. (__Saara__, __Trond__ This presupposes a discussion with Oslo. (__Trond__ and __Saara__ to continue this discussion) * For Lule Sámi: We would like to have a parallel corpus interface with NT (text only). This presupposes better quality texts (__Trond, Børre__) ** Better Lule NT text still not made. * The list of good candicates: The longest (admin) texts. ** We need a new option in ccat for analysing text while still keeping the xml tags and structure (__Tomi__) ** xml texts number

, preprocess finds <.> and ccat numbers them as . ** Then the aligner aligns... Top-two priorities: # Trond to discuss more with Lars on tag unification, and unify them. # Tomi to change ccat to be able to create the right input for the corpus analysis. # Lars to add text to the server. !!Language recognition TODO: * refine language recognition (__Saara__) ** in progress, continue discussion in [249|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=249] * create a short word list to help the trigram heuristics ** Trond has made such lists for all lgs except smj. * send __Saara__ smj files (__Trond__) * Add some flag to write into the xsl file (__Saara__): ** method:do not run lg recognition ** method:Choose between these 2: nob, sme, etc. !!!6. Infrastructure !!Paradigm generation [Greenland's language secretariat|http://www.oqaasileriffik.gl/dk/] has a paradigm generator based upon Xerox tools, we have asked for their source code, and will get it (site in Greeenlandic, English and Danish). Try e.g. illu "house", aput "snow" (or any word in st/kal/src/noun-kal-lex.txt) TODO: * Put __Saara__ and __Tero__ in contact with each other (__Trond__) * The paradigm generator should also have an xml-out option (for use in e.g. electronic dictionaries) - on hold till we know more about the generator !!Aligner TODO: * Read documentation and try out, give feedback to Bergen. (__Trond__, __Saara__) * __Saara__ to install the aligner, everyone to read the documentation ** done, waiting for the test files from Bergen. * __Trond__ and __Saara__ will continue this issue. !!Hyphenator __Trond__ and __Thomas__ have been updating the propernoun file with ^ tags. We need the tag in front of compound parts beginning in a vowel or in two or more consonants. Compound parts beginning with one consonant are handled correctly. TODO: * correct the treatment of hyphenation of word boundaries and exceptions (fst gymnastics) (__Sjur, Trond__) ** Still not done. * add a possibility to upload whole documents for hyphenation (and also analysis, generation, etc) (__Saara__) ** moved to Bugzilla * we should log all and every word/text uploaded/hyphenated/analyzed etc ** we'll do it, but it does not have first priority (__Saara__) *** moved to Bugzilla !!!7. Linguistics !!General - hyphenation See discussion, open questions and decission in the [previous meeting memo.|/doc/admin/weekly/2006/Meeting_2006-04-03.html] We did a sme overkill: V^CV, when the rulebased hyphenator gave the right result. For sme: Find the default, only tag the exceptions. (issue: VCsCV, VsCV) TODO: * Set up the mechanism for the hash transducer package - fst gymnastics, see above. * add exceptions marks to the {{smj}} lexicon (boks^távva) ** Status? We discussed with Kåre Tjihkkom, and the principles are clear, even behaviour around -st-, but it is not implemented in the lexicon !!North Sámi There are some heavy bugs (11 sme bugs all in all): * he(a)ddjiid is one * compounding cleanup - no shortening when normative, still shortening when analysing corpus texts? * vowel-shortening when compounding (we need the input from Pekka!) We should have some linguistic workshops while Maaren is here. TODO: * set up a linguistic workshop while Maaren is in Tromsø (and remember that she is on sick leave all the time) (__Trond, Thomas__) !!Lule Sámi TODO: * 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks * There are some open issues in the marginal area of the smj transducer: ** numerals, e.g. These become more visible as we get real texts. ** names ** compounds? Shortening here as well, but not in written language (some lexicalised exceptions, as in sme) ** loanwords? !!!8. Name lexicon infrastructure TODO: # refactor and prepare risten.no for multiple collections: ## refactor the code into more and more specific components according to our folder hierarchy (__Tomi, Sjur__) ### done for editors, some open issues for regular searches # write down the most common editing scenarios (to be used as guides for making the editing interface) (adding / changing ) (__Trond, Tomi__) ## Done. # develop the needed XQueries and interface (__Sjur, Tomi__) ## developing # data synchronisation between risten.no and the cvs repo (__Tomi__) ## nothing this week # test and review when ready # Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org, or the Lyndi England issue) (__Trond__) !!!9. Spellers Nothing until the new proper noun lexicon is in place. We don't have enough people to do both. Here's our most important targets for spellers in the near future: * aspell * hunspell !!!10. Public tender TODO: * read the offers (__Børre, Sjur, Trond, Tomi__) ** done * meet on Tuesday 13 to sum up the findings (__Børre, Sjur, Tomi, Trond__) ** done, need more info - questions sent * telephone meeting next Wednesday with Finnut (__Børre, Sjur__) ** postponed till we have more info. * new meeting on Thursday 18.5. at 10.00 AM (__Børre, Sjur, Tomi Trond__) !!!11. Other !!Summer vacation || Who || When | Børre | ? | Linda | ? | Maaren | ? | Saara | July | Sjur | ? | Thomas | 3.7 - 7.8 | Trond | July | Tomi | 8.7 - 16.7, more? !!Bug fixing __45__ open Divvun/Disamb bugs, and __25__ risten.no bugs Guess: 1/3 of the bugs are fixed already (?) Please help __Saara__ with bug 279 (Perl locale). Not much help... __Saara__ will contact __Roy__ on this issue. After the corpus issues have been somewhat settled, we should do a bug barnraising. ... and then a new one after the name lexicon is fixed. !! Move to victorio Trond's victorio problem: {{{ sme: FirstTag...1, ProperNoun...10000... *** Warning: Ignoring info strings. 20000...30000...flex scanner jammed make: *** [sme/bin/sme.save] Error 2 smj: Reading from 'smj/src/adv-smj-lex.txt' adv...1, Adverb...1064 Reading from 'smj/src/noun-smj-lex.txt' NounRoot...flex scanner jammed make: *** [smj/bin/smj.save] Error 2 }}} sma, kal compile without jamming.cd Saara compiles sme without problems, has problems with smj. Conclusion: it is a source code problem. Tomi and Børre are compiling smj just fine. !!Gobby 0.3 is working fine on Mac, Linux and Windows. Should be installed on all computers c.f. [http://darcs.0x539.de/trac/obby/cgi-bin/trac.cgi/wiki/InstallationGuide] (our preinstalled Xcode veriosn is 2.0, must be 2.1): * Børre - todo * Maaren - Børre to do it * Saara - todo * Sjur - ok * Thomas - Børre to do it * Tomi - todo * Trond - ok Trond should ask Lars Nygård, Per Langgård and Tero Avellan to install Gobby as well. !!!12. Summary, task list !! Børre * corpus work: ** send out contracts with accompanying letter ** Gather public texts, preferrably also parallel ones ** Continue converting text from input format to our xml ** convert nob and nno bible texts to be used as part of a parallel corpus ** review the paratext2xml converter ** convert smj NT to paratext ** Send out letters to the rest of the Iđut authors ** call Brita Kåven ** contact Kåfjord ** create weekly cron job to mirror Odin URL and detect new/updated pages ** Check the status & license of the corpus texts ** contact Korhonen & Kuhmunen * public tender: ** continue public tender offer evaluation ** meeting on Thursday 18.5. at 10.00 AM with __Sjur, Tomi, Trond__ ** telephone meeting with Finnut * install latest SEE * install Gobby using Darwin Ports (also for Thomas and Maaren) * move to Bugzilla: ** write docu for how to apply for a corpus user account * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Maaren * On sick leave !! Saara * Create a parallel corpora of the new testaments. * add more texts to the graphical corpus interface * grammatical searchability in the graphical corpus interface * Implement links to parallel files in corpus header. * Implement parallel corpus upload in web upload script * Check the status & license of the corpus texts and change the default of the availability to "license" instead of "free" * Rerun corpus conversion * Install Gobby * make xsl conversion routine for the typographic tags * update the corpus script(s) to only copy texts to the free/ dir which are explicitly marked as free * Add some language recognition flags to write into the xsl file * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Sjur * public tender: ** read & evaluate received offers ** meeting on Thursday 18.5. at 10.00 AM with __Børre, Tomi, Trond__ ** telephone meeting Friday with Finnut * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer * name lexicon: ** implement editing functions ** write down the most common editing scenarios, to guide development ** finalise refactoring for multiple collections (regular search interface) * update corpus-summary processing to adhere to the new structure * send bug report to Apple re filename matching and accented characters in Terminal * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Thomas * correct hyphenation of exceptions * correct hyphenation of smj -st- * work on compounding and derivation * smj G3 issue * sme G3 issue * set up a linguistic workshop while Maaren is in Tromsø !! Tomi * __move to Bugzilla:__ ** aspell UTF-8 suffix bug ** Document aspell infrastructure: finish doc/proof/spelling/X-spell/aspell.xml (it's almost done, but there are a couple of loose ends) * new proper name lexicon ** implement data synchronisation of proper nouns between risten.no and CVS ** XQuery refactoring and code development for our proper noun editor ** new version of xml2lexc (based on ccat), should handle complex names correct: construct entries like we have now from the different parts of a complex name entry * read aligner docu, install, provide feedback * install and test Gobby * Set up the mechanism for the hash-mark transducer package * meeting on Thursday 18.5. at 10.00 AM with __Børre, Sjur, Trond__ * add ccat option to analyse text while keeping the xml tags and structure * [fix bugs!|http://giellatekno.uit.no/bugzilla] !! Trond * better smj NT text * get fin, swe, nob and nno NT and OT in __paratext__ format * install aligner, test it and give feedback * fst gymnastics to add hyphenation and word boundary marks to hyphenation transducer * get/upgrade keys for __Børre's__ room for __Tomi__ and __Thomas__ * Rethink the doubletagging procedure for names, consider grammatically motivated semtag conversion routines ("Helsinki" from Plc to Obj to Org) * Check the status & license of the corpus texts * Work on the graphical corpus tag list * send __Saara__ smj files for language recognition * Put __Saara__ and __Tero__ in contact with each other * meeting on Thursday 18.5. at 10.00 AM with __Børre, Sjur, Tomi__ * ask __Lars Nygård, Per Langgård__ and __Tero Avellan__ to install Gobby 0.3 * set up a linguistic workshop while Maaren is in Tromsø * [fix bugs!|http://giellatekno.uit.no/bugzilla]. !!!13. Next meeting, closing 22.05.2006 09:30 Closed at 11:30