!!!Meeting setup
* Date: 30.01.2006
* Time: 09.30 Norw. time
* Place: Wherever we are :-)
* Tools: iChat, SubEthaEdit
!!! Agenda
# Opening, agenda review
# Reviewing the task list from two weeks ago
# Documentation - divvun.no
# Corpus gathering
# Corpus infrastructure
# Linguistics
# name lexicon infrastructure
# Other issues
# Summary, task lists
# Closing
!!!1. Opening, agenda review, participants
Opened at 09:48.
Present: __Børre, Sjur, Tomi, Trond__
Absent: __Maaren, Saara, Thomas__
Main secretary: __Børre__
Agenda accepted as is.
!!!2. Reviewing the task list from the last meeting
!! Børre
* send out contracts with accompanying letter
** Not done
* Gather public texts, preferrably also parallel ones
** Not done
* Contact Odin editor (__Ove Sæth__) to ask for source (and parallel) documents
** Done
* Continue converting text from input format to our xml
** Not done
* review code and documentation for corpus xsl files under version control
** Not done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
** Not done
!! Maaren
* work with risten.no
* discuss with relevant people regarding seminar on proofing tools, normativity
and SGL in February/March, including place.
!! Saara
* Convert the name lexicon from present format to xml for testing; final
conversion pending the editing fascilities
** done
* Refine language detection for Finnish
** not done
* Finish the review of the hyphenation detection.
** not done
* Review the handling of xsl-files in corpus infrastructure, including version
control
** in progress
* Do some testing for bug
[#211|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=211]
** not done
* optimize the preprocess script
** not done
* Write/update user documentation for the corpus usage in preparation for the
review in the project meetings next week.
** done
* finalize an improved working version of the CGI and command line scripts for
corpus additions
** in progress
* xml2lexc update to handle complex names: construct entries like we have now
from the different parts of a complex name entry
** tomi will do the xml2lexc-script(?)
* update conversion from lexc to xml (proper names) with the latest
refinements
** not done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!! Sjur
* Follow up the lawyer treatment of the contracts
** not done
* Project seminar
** plan and make schedule with __Trond__
*** done
** check which hotels SD has an agreement with
*** done
** plan XQuery/XSLT training session
*** done
*** the whole seminar is done
* Lule Sámi twol problems, with __Thomas__ and __Trond__
** not done
* follow up on voice group-chat not working to Sámediggi
** Test Marratech when the new Marratech server is in place
*** not done
* project planning with __Trond__, continued
** also look at the development processes - specification and testing
*** not done
* Follow up on place names from Norge Digitalt
** write an e-mail to or call __Bjørn Olav Megard__
*** not done
* Evaluate SFST as speller (and analyzer) lexicon
** more thorough analysis than was possible in Guovdageaidnu
*** not done
* write a background document on the corpus contracts
** not done
* continue proper name lexicon work and discussion
** done a lot at the seminar
* public tender:
** review offer from [Finnut Consult AS|http://www.finnut.no/]
*** done, as well as asked for two other offers, then picked one (Finnut), and
had a meeting with them, planning the whole PT process
* smj G3 issue with __Thomas__ and __Trond__
** not done
* sme G3 issue with __Thomas__ and __Trond__
** not done
* call EDD/__Christian Emil Ore__ about national place name lexicon
** not done
* risten.no/name lexicon development: fix bugs, continue development
** done some, backed up and restarted the server; the backup isn't completely
valid, due to some strange bug/conflict between the version on the server
and the version of the client. The XML files are ok though.
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!! Thomas
On sick leave
!! Tomi
* Aspell: Continue working on the affix file & aspell
** Contact aspell author (UTF-8 thing)
*** Not done
* corpus infrastructure:
** dtd location (both public and internal)
*** Not done
** cgi-admin script for adding xsl-files
*** Not done
* Document aspell and corpus infrastructure
** Not done
* Specification for new catxml in C++
** install and announce new ccat tool
*** Done
* new proper name lexicon
** remove last part of complex names not used as simplex names
*** Not done
** start looking at conversion of the name lexicon from present format to xml
*** Not done
** discuss the new lexicon format in the newsgroup
*** Not done
** Look into synchronisation of proper names with risten.no
*** Not done
** meeting to arrive at final xml format
*** Participated the meeting
** new version of xml2lexc (based on catxml, now ccat)
*** Not done
* hyphenation in corpus docs
** Not done, but has __Saara__ done it? Sjur: yes.
* comment review template made by __Saara__
** Not done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
** Not done
* pick up backpacks after Xmas
** Done
!! Trond
* Contact Odin editor (__Ove Sæth__) immediately to reopen contacts
** Done.
* Project seminar
** plan and make schedule with __Sjur__
** Done.
** check with Linda and Ilona whether we can start on Monday after lunch
** Done.
* sign contract with Bibelselskapet for Norwegian parallel texts
** Done.
* document corpus infrastructure, your part
* review corpus usage documentation (ccat)
** Done.
* discuss the new lexicon format in the newsgroup
** Done, but not extensively.
* smj G3 issue with __Sjur__ and __Thomas__
** Not done
* sme G3 issue with __Sjur__ and __Thomas__
** Not done
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
** I think we made some progress on some of the number bugs, but we haven't
looked at the bugzilla base yet.
!!!3. Documentation
!!Reviews
!ccat review
__Saara, Linda, Ilona, Trond__
Conducted the review as part of the seminar, although Thomas and Maaren weren't
there. Outcome:
It works mostly as documented, a few glitches were found and corrected. The
documentation is ''terse'' but concise (simple is beautiful, that is, in full
accordance to KISS). ccat -h gives:
{{{
-a Print all text elements.
-p Print plain paragraphs. (default)
-T Print paragraphs with title type.
-L Print paragraphs with list type.
-t Print paragraphs with table type.
-r
Recursively process directory dir and subdirs enountered.
-h Print this help message.
}}}
* (Trond:) I thought that -T printed titles, but it gave the same output as -a.
* (Tomi:) No, it gave you all regular paragraphs + titles, but no tables or
lists. The latest version will give you ''only'' titles, lists, etc according
to the specified option.
* (Sjur:) I would like a -v option, so we are able to identify which version to
send a bug report for :-) The version number should be autoincremented on each
CVS check-in/build, or manually set?
The [xml-based documentation|/doc/ling/catxml.html] was not completely
up-to-date with the latest changes, fixed in the meeting.
!!Other documentation
* __Børre__: Informed about the forrest documentation: the documentation tree
structure, how to add documents and get them into the menu.
!!!4. Corpus gathering
Discussed briefly how the formalities should be implemented: signature, Websak,
posting etc.
!!Collecting
See the [previous meeting memo|Meeting_2006-01-16] for what's to be done.
TODO: Still a ''lot'' for __Børre!__
!!Odin
DONE: __Trond__, and then __Børre__ to call __Ove Sæth__ to re-establish
contact.
__Sæth__ to discuss with colleagues about how to implement the cooperation.
!!Bible texts
{{{
ccat -t zcorp/gt/sme/bible/ot/1Mos_09-01.doc.xml | less
}}}
This gives everything.
What we want is to make a file-specific version of testament.xml, with these
properties:
* The first column should be suppressed
* The second column should be marked number or something
* The third column should be marked header if the typographic code in the fist
column indicates it is a header (1 or 3) and it should be marked text if the
typographic code indicates it is text (is 5).
{{{
sme$ls zcorp/orig/sme/bible/ot/
1Mos_09-01.doc Salmmat-_garvasat_0203.doc
}}}
There are two new books in the paratext format waiting in the nob orig and nno
orig directories.
TODO:
* write a paratext2xml converter
* convert smj NT to paratext
* ask to get fin and swe NT and OT in paratext format
We already have an embryonic converter: gt/script/testament.xsl
Usage: xsltproc /path/to/testament.xsl bible-text.xml > converted-text.xml
format.
!!!5. Corpus infrastructure
Task list:
# Include the xsl files under version control
## RCS version control is almost finished, but an issue with access control is
still open. Discussed a bit in the meeting, but nothing conclusive. We'll
continue the discussion in the newsgroup.
# Incorporate language detection as part of the corpus processing (__Saara__)
## Almost finished. Needs improved Finnish language model - presently it isn't
able to distinguish Finnish from Sámi (proving the family bonds:-)
# we need to review whether only automatic hyphen detection is good enough, or
whether manual post-processing in some form is needed. Delayed until we have
some results to base the review on.
## Acceptable results: 90% of all real hyphens correctly tagged.
# CGI-admin script to add xsl-file to a corpus file that doesn't have one
(__Saara__)
Things are moving forward, but still more work to do. The list is left as is.
!!!6. Linguistics
Nothing today, our linguists are on sick leave or not participating. For the
tasks and their status, see the [previous meeting memo|Meeting_2006-01-16].
!!!7. Name lexicon infrastructure
!!Complex names
Task list for this issue:
* make sure xml2lexc can handle complex names in ways compatible with our
present tool chain (=reconstruct the lexc format we have now) (__Tomi__)
** the resulting file format should be identical to our present prop-name
file (=lexc), that can then be converted to our new xml format using the same
script as for the regular names (__Tomi__ or __Saara__, but only when the
technical details are settled)
* __Saara__ has added the analyzer as part
of the preprocess, but it is slow, and needs to be optimized.
!!XML format
Tasks:
# make a test lexicon for evaluating the format, set up the editing, and test it
(__Saara__)
## Done
# update conversion from lexc to xml to reflect new xml format (__Saara__)
## mostly done, some open questions left
# testing of conversion
# eXist as editor:
## develop the needed XQueries and interface
## synchronisation between risten.no and
## test whether eXist as editor is actually working well
!!!8. Other
!!SGL Seminar
* SGL/normativity seminar
** all members = potentially/likely all languages
*** not all languages, only North Sámi
** date? As early as possible, end of February/beginning of March
** place? __Maaren__ will investigate
!!Technical issues
* The mac os / perl bug (at least __Trond__ and __Sjur__ has it, [Bugzilla
#211|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=211]):
** utf8 "\xC4" does not map to Unicode at /Users/trond/gt/script/preprocess line
82. This msg did not show up in 10.3 (perl 5.8.1), but does so in 10.4 (perl
5.8.6). It is probably a perl - OS mismatch. (__Trond__, __Saara__,
__Tomi__)
*** 10.4 introduced support for locales in the shell (10.3 and earlier didn't
know about locales)
** Test: the result of the last line should indicate whether this is a problem
in cat or a Perl/OS mismatch.
** Is this a problem with ccat?
*** It doesn't seem so (3 min and still counting)
*** In the end, the bug turned up with ccat as well. I gave the command:
*** zcorp/gt/sme/*/*xml | preprocess --abbr=bin/abbr.txt | lookup -flags
mbTT -utf8 bin/sme.fst | lookup2cg | vislcg --grammar=src/sme-dis.rle
--minimal | sort | less
*** and it (in the end) responded:
{{{
1729 constraint rules
utf8 "\xA1" does not map to Unicode at /home/trond/gt/script/preprocess line 109, <> chunk 12.
}}}
To ccat's defence I must say that cat, in a similar situation, would have given far
more error messages (hold on, testing still under way).
{{{
preprocess file_name.txt - OK
cat file_name.txt | preprocess - bug!!
catxml file_name.xml | preprocess - ??
ccat filename | preprocess - bug !!
}}}
This bug isn't a high priority any more, because ccat behaves differently than
cat, and because there is the possibility of avoiding cat when working locally.
BUG: close as Won't fix.
!!Bug fixing
__30__ open bugs (and 25 risten.no bugs)
!!Norwegian ispell press release
The i18n section of Skolelinux plans a press release including a paragraph
about our project. We will ask them to reformulate a couple of things, and
remove the links they've included. Text submitted to them:
* Dette er den eneste kilden til retteprogram for norsk uavhengig av Microsoft,
bl.a. det som brukes av Apple.
* Et separat prosjekt ved Sametinget er i gang for å utvikle samiske
retteprogram, bl a for Linux. Se http://www.divvun.no for mer info.
!!!9. Summary, task list
!! Børre
* send out contracts with accompanying letter
* Gather public texts, preferrably also parallel ones
* Continue converting text from input format to our xml
* review code and documentation for corpus xsl files under version control
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!! Maaren
* work with risten.no
* discuss with relevant people regarding seminar on proofing tools, normativity
and SGL in February/March, including place.
!! Saara
* continue discussion on the new lexicon format
* Refine language detection for Finnish
* Finnish the review of the hyphenation detection.
* Review the handling of xsl-files in corpus infrastructure, including version
control
* Do some testing for bug
[#211|http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=211]
* Fix the preprocess script and optimize it by building an analyzator
for the multi-part expressions.
* finalize an improved working version of the CGI and command line scripts for
corpus additions
* update conversion from lexc to xml (proper names) with the latest refinements
* Try to add numeral treatment as part of the analyzator.
* Change character coding detection to paragraph-based.
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!! Sjur
* Follow up the lawyer treatment of the contracts
* Lule Sámi twol problems, with __Thomas__ and __Trond__
* follow up on voice group-chat not working to Sámediggi
** Test Marratech when the new Marratech server is in place
* project planning with __Trond__, continued
** also look at the development processes - specification and testing
* Follow up on place names from Norge Digitalt
** write an e-mail to or call __Bjørn Olav Megard__
* Evaluate SFST as speller (and analyzer) lexicon
** more thorough analysis than was possible in Guovdageaidnu
* write a background document on the corpus contracts
* continue proper name lexicon work and discussion
* public tender:
** review offer from [Finnut Consult AS|http://www.finnut.no/]
* smj G3 issue with __Thomas__ and __Trond__
* sme G3 issue with __Thomas__ and __Trond__
* call EDD/__Christian Emil Ore__ about national place name lexicon
* risten.no/name lexicon development: fix bugs, continue development
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!! Thomas
* work on North Sámi compounding and derivation
* review corpus usage documentation
* smj G3 issue with __Sjur__ and __Trond__
* sme G3 issue with __Sjur__ and __Trond__
!! Tomi
* Aspell: Continue working on the affix file & aspell
** Contact aspell author (UTF-8 thing)
* corpus infrastructure:
** dtd location (both public and internal)
** cgi-admin script for adding xsl-files
* Document aspell and corpus infrastructure
* ccat: add a -v option - it should return the version of the tool
* new proper name lexicon
** remove last part of complex names not used as simplex names
** start looking at conversion of the name lexicon from present format to xml
** discuss the new lexicon format in the newsgroup
** Look into synchronisation of proper names with risten.no
** new version of xml2lexc (based on catxml, now ccat)
*** xml2lexc update to handle complex names: construct entries like we have now
from the different parts of a complex name entry
* comment review template made by __Saara__
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!! Trond
* Work on corpus texts with Børre.
* 3-part compounds with __Sjur__ and __Thomas__.
* smj G3 issue with __Sjur__ and __Thomas__.
* sme G3 issue with __Sjur__ and __Thomas__.
* [fix bugs!|http://giellatekno.uit.no/bugzilla]
!!!10. Next meeting, closing
06.02.2006 09:30
Closed at 12:03