!!!Meeting setup

* Date: 06.06.2006
* Time: 09.30 Norw. time
* Place: Where we are
* Tools: SubEthaEdit, iChat

!!!Agenda

# Opening, agenda review
# Reviewing the task list from two weeks ago
# Documentation - divvun.no
# Corpus gathering
# Corpus infrastructure
# Infrastructure
# Linguistics
# name lexicon infrastructure
# Spellers
# Other issues
# Summary, task lists
# Closing

!!!1. Opening, agenda review, participants

Opened at 09:59.

Present: __Sjur, Thomas, Trond, Tomi__

Absent: __Børre, Maaren, Saara__

Agenda accepted as is.

!!!2. Updated task status since last meeting

!! Børre
* corpus collection:
** send out contracts with accompanying letter
*** Maaren has signed a contract, which is sent to Davvi Girji
** Gather public texts, preferrably also parallel ones
*** Not done
** Continue converting text from input format to our xml
** convert nob and nno bible texts to be used as part of a parallel corpus
*** Not done
** review the paratext2xml converter
*** Not done
** convert smj NT to paratext
*** Not possible before we have another paratext NT.
** Send out letters to the rest of the IÄ‘ut authors
*** Not done
** send contract to __Kurt Tore Andersen__
*** Not done
** call __Brita KÃ¥ven__ again
*** Haven't got in touch with her
** call __Harald Gaski__
*** Met with him. He has to send contracts to his writers union, to check
    if the contracts are ok with them.
** contact __Ája__ (Kåfjord)
*** Not done
** Send renaming scripts to __R. Valkeapää__
*** Working on them
** complete __Min Áigi__ metadata
*** Halfway done
** call __BÃ¥rd Eriksen__
*** Not done
* corpus access:
** meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
*** Delayed once more
** set upp the unix group structure for corpus users (also __Saara, Trond__)
*** Waiting for outcome of above meeting
* set up Bugzilla automatic reminders for open issues; ask Thor-Øivind if needed
** Not done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Maaren
* On sick leave

!! Saara
* Create a parallel corpora of the new testaments
* add more texts to the graphical corpus interface
* grammatical searchability in the graphical corpus interface
* Implement links to parallel files in corpus header
** done
* Implement parallel corpus upload in web upload script
* Install Gobby
* set upp the unix group structure for corpus users (also __Børre, Trond__)
* Test the aligners once again
* make an analyser that retains the xml structure for Oslo
** in progress (with Tomi). The exact structure of the output still open
* discuss parallel text markup in newsgroup
** done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Sjur
* public tender:
** write e-mail to PL and ask for more clarification
*** done
* fst gymnastics to add hyphenation and word boundary marks to hyphenation
  transducer
** not done
* name lexicon:
** implement editing functions
** finalise refactoring for multiple collections (regular search interface)
*** some work on researching new ways of doing all-collection searches, and at
    the same time allow for different xml structures
* change corpus-summary processing to generate smaller pages
* meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
** postponed
* move to Bugzilla:
** proofed/unproofed parallel text issue
*** done
** xml output of paradigm generator
*** done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Thomas
* hyphenation-rule-set 
** a few bugs, otherwise finished
* work on compounding and derivation
** finished with derivation
* lule sámi incoming words
** added a few
* sme G3 issue
** nothing this week

!! Tomi
* new proper name lexicon
** data synchronisation of proper nouns between risten.no and CVS
*** not done
** XQuery refactoring and code development for our proper noun editor
*** not done
** new version of xml2lexc (based on ccat), should handle complex names correct:
   construct entries like we have now from the different parts of a complex name
   entry
*** not done
* read aligner docu, install, provide feedback
** not done
* Set up the mechanism for the hash-mark transducer package
** not done
* add ccat option to analyse text while keeping the xml tags and structure
** done some
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Trond
* better smj NT text
** Not done
* get fin, swe, nob and nno NT and OT in __paratext__ format
** Discussed with fin and swe societies, the Swedes are not quite satisfied with
   their paratext version, and prefer us to do the Word version. Børre and I
   should take a new round on this.
* install aligner, test it and give feedback
** Not done
* fst gymnastics to add hyphenation and word boundary marks to hyphenation
  transducer
** Not done
* Put __Saara__ and __Tero__ in contact with each other
** Per will send the PHP code. Details about Tero will be given at the meeting.
* meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
** Postponed.
* set upp the unix group structure for corpus users (also __Børre, Saara__)
** Postponed.
* [fix bugs!|http://giellatekno.uit.no/bugzilla].
** Worked on some bugs.

!!!3. Documentation

TODO:
* documentation on how to apply for a user account for the corpus repo
  (__Børre__)
** we will administer the corpus user accounts ourselves
** We first have to discuss and decide what we want before __Børre__ can write
   documentation (see further down)

!!!4. Corpus gathering

!!Collecting

See a [previous meeting memo|Meeting_2006-01-16] for what's to be done.

TODO:
* Send out the rest of the letters (__Børre__)

New contracts:
* none last 2 weeks

!!Olavi Korhonen's Lule Sámi dictionary.

TODO:
* set up user account/corpus access for Olavi (__Børre__)
** we need the infrastructure for corpus access in place first

!!KIO Grafisk and the IÄ‘ut books

__TODO__:
* send letter to Kurt Tore Andersen (__Børre__)
* send letters to the other authors (__Børre__)

!!Bible texts

The Swedish Bible Society is reluctant to give us the paratext version, as it is
linguistically inferiour. Instead they suggested a MS Word version of the text.

__Børre__ and __Trond__ suggest that we give the MS Word version a second try,
at least to be able to get hard facts about the problems with that format when
discussing with the swedes.

We have asked for ''any'' language version in paratext format from the Norwegian
Bible Society, to use as a reference when making the __smj__ paratext version.

We are still waiting for the Norwegian versions, but Finnish and Swedish are
within reach. Finnish is available in a newsgroup (text-only) format, and we 
have received MS Word of the  Swedish version. Børre has had some problems
converting the Word file, though.

TODO:
* get nob and nno NT and OT in __paratext__ format. (__Trond__)
* convert smj NT to paratext (__Børre__)
* convert fin, swe to paratext or directly to our XML (__Børre__)

!!Davvi Girji

Talked to Harald Gaski, he will send the contracts to the writers' organisation,
to ensure the contract is ok.

__TODO__:
* call Brita Kåven again towards the end of the week (__Børre__)
* call the authors (__Børre__)
** Harald Gaski
*** done

!!Min Áigi

__TODO__:
* complete metainformation (__Børre__)

!!KÃ¥fjord

Promised to send us texts. Some texts have arrived, but nothing from Ája.

__TODO__:
* contact Ája (__Børre__)
** Not done

!!Sámi Instituhtta

TODO:
* send renaming scripts to R. Valkeapää (__Børre__)

!!Čálliid Lágádus

__Børre__ talked to them, they are positive, and will give us print-ready pdf's.
[http://www.calliidlagadus.org/]

!!Árran

The negotiations are underway, they discuss it in a meeting today, and we have
an appointment for later today or tomorrow (both their internal texts, and
discussions on Báhko). All further negotiations will go through Bård Eriksen
/ Báhko.

TODO:
* call Bård Eriksen (__Børre__)

!!!5. Corpus infrastructure

!!General

Errors in the Antiword conversions found when parsing the xml corpus. Main
problem is to skip headers and footers - now they are included as part of the
text, and are intermingled within the regular text, e.g. in the middle of a
sentence.

TODO (all of these in priority order, the third option is really a last resort):
# fine-tune the initial conversion in antiword (__Børre__ or __Saara__)
# make file-specific fixes in the file-speciflc xsl file (by having our local
  xsl expert __read__ our corpus, and spot obvious omissions in the conversion
  (__Saara__)
# Manually fix the resulting files before sending off to analysis (??, this
  option should be postponed to just before the final, official release, and
  evaluated at that point)

!!User accounts and access

TODO:
* discuss and decide upon the exact access policy we want to give corpus users;
  meeting on Tuesday May 23, at 9.30 (__Børre, Sjur, Trond__)
** still not done, new date: Wednesday 14.6.2006, at 9.30
* set upp the unix group structure to open for a new category of users:
  Linguists with access to the texts on the servers. Some of
  __Børre, Saara, Trond__.

!!More texts to the graphical corpus interface:

We need to get the infrastructure complete to be able to do this, then it
should be a piece of cake.

Saaara has made corpus-analyze.pl, a script to analyse text while keeping the
xml structure. It is working, but the output format needs some tweaking.

TODO:
# refine xml-tagged output (__Saara__ and __Tomi__)
# add text to the server (__Lars__)

!!Aligner

__Trond__ and __Saara__ will continue this issue.

We need markup of parallelism in the corpus DTD, at least an indication of which
documents belong together. Discussion to continue in the newsgroup (__Saara__
has started it - please respond!).

!!Language recognition

Still waiting for more __smj__ text to improve it.

!!Free and non-free texts

Anything? Final check with __Børre__ and __Saara__ - waiting for them to return.

!!Corpus summary

TODO:
* trim generated corpus summary pages (__Sjur__)
** suggestion: lump together files with content less than X paragraphs (X < 5?)

!!Proofed vs unproofed corpus files

TODO:
* count the number of parallel files of type unproofed/proofread (__Saara__)
** done, 35 such file pairs
** delayed for the time being, move to Bugzilla (__Sjur__)
*** done

!!!6. Infrastructure

!!Paradigm generation

Goal: Reuse Greenlandic code for paradigm generation.

We have now received the original PHP code from Tero/Per. It seems quite easy to
adapt, although it will of course require work on our part.

TODO:
* convert or adapt the received PHP to our needs (__Tomi__ or __Saara__)
* move xml-out option to Bugzilla (__Sjur__)
** done

!!Hyphenator

__Thomas__ is finished with adding ^ tags to the __sme__ noun file.

__Trond__ and __Thomas__ have been working on the smj rule component, and have
improved both the treatment of weak grade consonant clusters (preconsonantal
geminates) and on some loan word patterns.

TODO:
* correct the treatment of hyphenation of word boundaries and exceptions (fst
  gymnastics) (__Sjur, Trond__)
** Still not done - postponed till next week (w 24)
* Update the sme and sma rule sets with the insights gained from smj updates.

!!Automatic Bugzilla reminder for untouched bugs

We need to get a summary by e-mail for all bugs not touched in more than 5(?)
weeks. Do we want it? Yes. It is possible? __Børre__ will look around, if
nothing found, he'll ask Thor-Øivind.

TODO:
* set up Bugzilla to send automatic reminders for bugs not touched in a given
  timeframe; ask Thor-Øivind for help if needed (__Børre__)

!!!7. Linguistics

!!Name double-tagging

Conclusion, in a principled fashion: 

# hardcoded sem-tags win
# There is a sem-tag conversion procedure: according to a hierarchy of sem-tags:
  Any Plc can be interpreted as Sur, etc. (to be spelled out)

TODO:
* when adding new names, only use one sem-tag unless there are known objects
  which belong to different sem classes(?) (__all linguists__)
* write disamb rules to implement the system above (__Trond, Linda__)

!!North Sámi

Topic: Actio+compound - how productive? How much does it destroy speller
performeance by being close to real spelling errors with a high editing distance
to the correct form?

{{{
Spelling error from typos.txt:
4       vuolggahansádji             vuolggasadji

vuolggahan
vuolggahan      vuolgga+N+Sg+Nom+Foc
vuolggahan      vuolggahit+V+TV+PrfPrc
vuolggahan      vuolggahit+V+TV+Ind+Prs+Sg1
vuolggahan      vuolggahit+V+TV+Actio+Acc
vuolggahan      vuolggahit+V+TV+Actio+Gen
vuolggahan      vuolggahit+V+TV+Actio+Nom

vuolggahansadji
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolgga+N+SgNomCmp#ho+N+SgNomCmp#atnu+N+SgNomCmp#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
vuolggahansadji vuolggahit+V+TV+Actio#sadji+N+Sg+Nom
}}}

TODO:
* investigate Actio compounding (__Thomas__)
* discuss findings with __Maaren__, and later all of us (__Thomas__)

!!Lule Sámi

There are some open issues in the marginal area of the smj transducer:
* numerals, e.g. Our poor treatment of number words becomes more visible 
  as we get real texts (guhttatuvsánanielljatjuodegålmmålåkvihtta is not
  recognised, etc.).
* names => waiting for the new name lexicon
* compounds? Shortening here as well, but not in written language (some
  lexicalised exceptions, as in sme) =>  nothing to discuss/implement
* loanwords? We should consider importing the ^LOAN words from sme and 
  Lule-ify them. Otherwise, loan words should come from corpora.

TODO:
* 50 unknown words left+2 abbr. +moaddi etc (numerals) need more checks
  (__Thomas__)
** progressing fine
* add proper numeral analysis/treatment (__Thomas__)
* add loanwords (e.g. latin -ere verbs) (__Thomas__)

!!!8. Name lexicon infrastructure

TODO:
# finish refactoring for multiple collections in the search interfarce
  (__Sjur__)
## progressing, investigating options
# develop the needed XQueries and interface (__Sjur, Tomi__)
## progressing, done some, commited
   (adding new term, create-termc-entry.xq)
# data synchronisation between risten.no and the cvs repo (__Tomi__)
## discussion started on eXist-list, nothing useful came up. We need to
   reformulate the question from our perspective, and bring it up again
   (__Sjur__)

!!!9. Public tender

We have received answers from LS, and are now wating for the PL answers. Their
deadline is the coming Thursday.

TODO:
* write clarification letters to PL (__Sjur__ with help from
  __Børre, Trond__)
** done
* evaluate finally the offers based on the answers to the clarification letters

!!!10. Other

!!Summer vacation

|| Who  || When
| Børre  | August
| Linda  | ?
| Maaren | ?
| Saara  | July
| Sjur   | at least 2 weeks in July, but still open
| Thomas | 3.7 - 7.8
| Trond  | 3.7 - 14.8 (last two weeks off at summer school)
| Tomi   | 8.7 - 16.7, 2 more weeks in July and/or August

!!Bug fixing

__43__ open Divvun/Disamb bugs (two down!), and __25__ risten.no bugs

Guess: 1/3 of the bugs are fixed already (?)

Please help __Saara__ with [bug
279|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=279]
 (Perl locale). Not much help...
__Saara__ will contact __Roy__ on this issue.

After the corpus issues have been somewhat settled, we should do a bug
barnraising. ... and then a new one after the name lexicon is fixed.

!!Gobby

Installed to most computers (only __Saara__ missing), now we need to test it in
a multiuser scenario.

__Trond__ has asked __Lars Nygård__ and __Tero Avellan__ to
install Gobby as well. __Per Langgård__ is already using it.

TODO:
* install Gobby (__Saara__)
* test it (__Tomi, Trond, Sjur, Børre, Lars?__)
* if successful, document its use within our project (__Børre__)

!!SEE 2.5 extensions

TODO:
* give the jspwiki syntax colouring mode to all (__Sjur__)
** JSPWiki.mode checked in to cvs in {{gt/src/}} - just double-click the mode!
* write a perl script to extract all TODO items and sort them according to
  responsible person; wrap the script in a AppleScript available from the menu
* other modes on the whish list:
** twol mode
** xfst mode
** lexc mode
** vislcg mode

!!!11. Next meeting, closing

19.06.2006 09:30

Sjur is away on Friday June 16.

Closed at 11:21.

!!!Appendix - task lists for the next week

!! Børre
* corpus collection:
** send out contracts with accompanying letter
** Gather public texts, preferrably also parallel ones
** Send out letters to the rest of the IÄ‘ut authors
** send contract to __Kurt Tore Andersen__
** call __Brita KÃ¥ven__ again
** contact __Ája__ (Kåfjord)
** Send renaming scripts to __R. Valkeapää__
** call __BÃ¥rd Eriksen__
* corpus conversion:
** Continue converting text from input format to our xml
** convert nob and nno bible texts to be used as part of a parallel corpus
** convert fin, swe to paratext or directly to our XML
** review the paratext2xml converter
** convert smj NT to paratext
** complete __Min Áigi__ metadata
* corpus access:
** meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
** set upp the unix group structure for corpus users (also __Saara, Trond__)
* set up Bugzilla automatic reminders for open issues; ask Thor-Øivind if needed
* test Gobby (with others)
* document use of Gobby within our project if above test is ok
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Maaren
* On sick leave

!! Saara
* Create a parallel corpora of the new testaments
* add more texts to the graphical corpus interface
* grammatical searchability in the graphical corpus interface
* Implement parallel corpus upload in web upload script
* Install Gobby
* set upp the unix group structure for corpus users (also __Børre, Trond__)
* Test the aligners once again
* refine the xml output of the xml-tagged analyses
* convert or adapt the received PHP for paradigm generation to our needs
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Sjur
* public tender:
** collect the last clarifying answers, and make a final proposal to the board
* fst gymnastics to add hyphenation and word boundary marks to hyphenation
  transducer
* name lexicon:
** implement editing functions
** finalise refactoring for multiple collections (regular search interface)
* change corpus-summary processing to generate smaller pages
* meeting 23.5. t 9.30: discuss and decide upon the exact access policy we want
* test Gobby (with others)
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Thomas
* investigate Actio compounding, first sme, later smj
* discuss findings with __Maaren__, and later all of us
* add proper numeral analysis/treatment to smj
* add loanwords (e.g. latin -ere verbs) to smj
* work on compounding
* lule sámi incoming words
* bug-fixing
* sme G3 issue

!! Tomi
* new proper name lexicon
** data synchronisation of proper nouns between risten.no and CVS
** XQuery refactoring and code development for our proper noun editor
** new version of xml2lexc (based on ccat), should handle complex names correct:
   construct entries like we have now from the different parts of a complex name
   entry
* read aligner docu, install, provide feedback
* Set up the mechanism for the hash-mark transducer package
* refine the xml output of the xml-tagged analyses
* convert or adapt the received PHP for paradigm generation to our needs
* test Gobby (with others)
* [fix bugs!|http://giellatekno.uit.no/bugzilla]

!! Trond
* better smj NT text
* get fin, swe, nob and nno NT and OT in __paratext__ format
* install aligner, test it and give feedback
* fst gymnastics to add hyphenation and word boundary marks to hyphenation
  transducer
* meeting 9.6. t 9.30: discuss and decide upon the exact access policy we want
* set upp the unix group structure for corpus users (also __Børre, Saara__)
* test Gobby (with others)
* [fix bugs!|http://giellatekno.uit.no/bugzilla].