Oc @ s? d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l Z d d% d YZ
d e j f d YZ d e j f d YZ
d
d& d YZ d d' d
YZ d e j f d YZ d d( d YZ d e j f d YZ d e f d YZ d e j f d YZ d d) d YZ d e j f d YZ d d* d YZ d e j f d YZ d d+ d! YZ d" d, d# YZ e d$ k r;x[ e
e e e e e e g D]; Z e j Z e j e j e e j! j" e qWn d S(- iN( t etree( t doctestcomparet ParallelFilec B sV e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
RS( sY
A class that contains the info on a file to be parallellized, name and language
c C s | | _ | | _ | j d S( N( t namet paralangt setLang( t selfR R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt __init__' s c C s | j S( s6
Return the absolute path of the file
( R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getName, s c C s t j j | j S( s0
Return the dirname of the file
( t ost patht dirnameR ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt
getDirname2 s c C s t j j | j S( s1
Return the basename of the file
( R R
t basenameR ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getBasename8 s c C s/ t j | j } | j j d | _ d S( s*
Set the lang of the file
s* {http://www.w3.org/XML/1998/namespace}langN( R t parseR t getroott attribt lang( R t
origfile1Tree( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR > s c C s | j S( s*
Get the lang of the file
( R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getLangE s c C sf t j | j } | j } | j d } x/ | D]' } | j d | j k r7 | j d Sq7 Wd S( sf
Get the basename of the parallel file
Input is the lang of the parallel file
s .//parallel_texts* {http://www.w3.org/XML/1998/namespace}langt locationN( R R R R t findallR R ( R R t roott
parallelFilest p( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getParallelBasenameK s
c C sD | j j | j | j } | j d } t j j | | S( s>
Infer the absolute path of the parallel file
s .xml( R t replaceR R R R R
t join( R t parallelDirnamet parallelBasename( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getParallelFilenameW s !( t __name__t
__module__t __doc__R R R R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR " s t TestParallelFilec B sM e Z d Z d Z d Z d Z d Z d Z d Z d Z RS( s1
A test class for the ParallelFile class
c C s! t t j d d d | _ d S( Nt GTFREEsA /prestable/converted/sme/facta/skuvlahistorja2/aarseth2-s.htm.xmlt nob( R R t environt pfile( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt setUpd s c C s | j | j j d d S( Ns aarseth2-s.htm.xml( t assertEqualR' R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testBasenameg s c C s4 | j | j j t j j t j d d d S( NR$ s. /prestable/converted/sme/facta/skuvlahistorja2( R) R' R R R
R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testDirnamej s c C s( | j | j j t j d d d S( NR$ sA /prestable/converted/sme/facta/skuvlahistorja2/aarseth2-s.htm.xml( R) R' R R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testNamem s c C s | j | j j d d S( Nt sme( R) R' R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testLangp s c C s | j | j j d d S( Ns aarseth2-n.htm( R) R' R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGetParallelBasenames s c C s( | j | j j t j d d d S( NR$ sA /prestable/converted/nob/facta/skuvlahistorja2/aarseth2-n.htm.xml( R) R' R R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGetParallelFilenamev s (
R R! R" R( R* R+ R, R. R/ R0 ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR# ` s t TestSentenceDividerc B sM e Z d Z d Z d Z d Z d Z d Z d Z d Z RS( s/ A test class for the SentenceDivider class
c C s t d d | _ d S( Ns4 parallelize_data/finnmarkkulahka_web_lettere.pdf.xmlR- ( t SentenceDividert sentenceDivider( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR( | s c C s t j | d t } t j | d t } t j } | j | | d s | j t j d | | d j d } t
| n d S( s5
Check if two xml snippets are equal
t pretty_printi t s utf-8N( R t tostringt TrueR t LXMLOutputCheckert check_outputt output_differencet doctestt Examplet encodet AssertionError( R t gott wantt
string_gott string_wantt checkert message( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt assertXmlEqual s *c C sL | j | j j d | j | j j d | j | j j j j d d S( sA Check that the constructor makes what it is suppposed to
i R- t _ElementTreeN( R) R3 t sentenceCountert docLangt
inputEtreet __class__R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testConstructor s c C s< | j j | j j } t j d } | j | | d S( NsA parallelize_data/finnmarkkulahka_web_lettere.pdfsme_sent.xml.test( R3 t processAllParagraphst documentR R RE ( R R? R@ ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testProcessAllParagraphs s
c C s d | j _ t j d } | j j | } t j d } | j | | t j d } | j j | } t j d } | j | | t j d } | j j | } t j d } | j | | d | j _ t j d } | j j | } t j d
} | j | | t j d } | j j | } t j d } | j | | t j d
} | j j | } t j d } | j | | d S( s0 Check the output of processOneParagraph
R- s Jápmá go sámi kultuvra veahážiid mielde go nuorat ovdal guldalit Britney Spears go Áilluhačča? máksá Finnmárkkuopmodat. § 10 Áššit meahcceeatnamiid
s Jápmá go sámi kultuvra veahážiid mielde go nuorat ovdal guldalit Britney Spears go Áillohačča ?máksá Finnmárkkuopmodat .§ 10 Áššit meahcceeatnamiid
sZ Stuora oassi Romssa universitehta doaimmain lea juohkit dieđuid sámi, norgga ja riikkaidgaskasaš dutkanbirrasiidda, sámi ja norgga eiseválddiide, ja sámi servodagaide (geahča mielddus A “Romssa universitehta ja guoskevaš institušuvnnaid sámi dutkan ja oahpahus” álggahusa ).
s= Stuora oassi Romssa universitehta doaimmain lea juohkit dieđuid sámi , norgga ja riikkaidgaskasaš dutkanbirrasiidda , sámi ja norgga eiseválddiide , ja sámi servodagaide ( geahča mielddus A “ Romssa universitehta ja guoskevaš institušuvnnaid sámi dutkan ja oahpahus ” álggahusa ) .
sA Artikkel i boka Samisk skolehistorie 2 . Davvi Girji 2007.
s] Artikkel i boka Samisk skolehistorie 2 .Davvi Girji 2007 .
R% si Bjørn Aarseth med elever på skitur - på 1950-tallet. (Foto: Trygve Madsen)
ss Bjørn Aarseth med elever på skitur - på 1950-tallet .( Foto : Trygve Madsen )
sa finne rom for etablering av en fast tilskuddsordning til allerede etablerte språksentra..
so finne rom for etablering av en fast tilskuddsordning til allerede etablerte språksentra .
s6 elevene skal få! Sametingsrådet mener målet
sR elevene skal få !Sametingsrådet mener målet
N( R3 RH R t XMLt processOneParagraphRE ( R R R? R@ ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testProcessOneParagraph s4 c C sP d | j _ t j d } | j j | } t j d } | j | | d S( s4 Test how SentenceDivider handles quotemarks
R% s Forsøksrådet for skoleverket godkjente det praktiske opplegget for kurset i brev av 18/8 1959 og uttalte da bl. a.: «Selve innholdet i kurset virker gjennomtenkt og underbygd og ser ut til å konsentrere seg om vesentlige emner som vil få stor betydning for elevene i deres yrkesarbeid. Med flyttsame-kunnskapen som bakgrunn er det grunn til å vente seg mye av dette kursopplegget.»
s Forsøksrådet for skoleverket godkjente det praktiske opplegget for kurset i brev av 18/8 1959 og uttalte da bl. a. : « Selve innholdet i kurset virker gjennomtenkt og underbygd og ser ut til å konsentrere seg om vesentlige emner som vil få stor betydning for elevene i deres yrkesarbeid . Med flyttsame-kunnskapen som bakgrunn er det grunn til å vente seg mye av dette kursopplegget . »
N( R3 RH R RO RP RE ( R R R? R@ ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testQuotemarks s
c C sR | j j d d d d d d g } | j | j d d | j | j d d S(
Nu Sámerievttiu ovdáneapmiu leau dahkanu vuđđosau Finnmárkkuláhkiit idt 0u@ Sámerievtti ovdáneapmi lea dahkan vuđđosa Finnmárkkuláhkii( R3 t makeSentenceR) R t text( R t s( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testMakeSentence s $(
R R! R" R( RE RK RN RQ RR RX ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR1 y s # R2 c B sD e Z d Z d Z d Z d Z d Z d Z d Z RS( s& A class that takes a giellatekno xml document as input.
It spits out an xml document that has divided the text inside the p tags
into sentences, but otherwise is unchanged.
Each sentence is encases in an s tag, and has an id consisting of the
outputfilename and a numberic
c C s( d | _ | | _ t j | | _ d S( Ni ( RG RH R R RI ( R t inputXmlfileR ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C sh t j d | _ t j d } | j j | x0 | j j d D] } | j | j | qD Wd S( sL Go through all paragraphs in the etree and process them one by one.
RM t bodys //pN( R t ElementRM t appendRI R RP ( R RZ t paragraph( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRL s
c C sN t | d } t j | j } | j | d t d d d t | j d S( s0 Write the given tmx to the outfile name
t wR4 t encodings utf-8t xml_declarationN( t openR t ElementTreeRM t writeR7 t close( R t outfilet ft et( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt writeResult s c C s& g } | j d k rD t j j t j d d } d d | g } nO t j j t j d d } t j j t j d d } d d | d | g } t j | d t j d
t j d t j } | j | j d \ } } | j
d
k r t j d IJt j | IJt j | IJt j
n | Sd S( s} Send the text in preprocessInput into preprocess, return the result.
If the process fails, exit the program
R% t GTHOMEs st/nob/bin/abbr.txtt
preprocesss --abbr=s gt/sme/bin/abbr.txts gt/sme/bin/corr.txts --corr=t stdint stdoutt stderrs utf-8i s Could not divide into sentencesN( RH R R
R R& t
subprocesst Popent PIPEt communicateR= t
returncodet sysRm t exit( R t preprocessInputt preprocessCommandt abbrFilet corrFilet subpt outputt error( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getPreprocessOutput s *
c
C s t j d } | j d } d j | } | r| j | } g } t } d } x| j d D]} | j | j d | d k r | t k r t } q t } | d k s | d k r | j | j
| g } q n | d k s| d k s| d k r`| t k r`| d g k rW| d g k rW| d g k rW| j | j
| n g } n | } qd Wt | d
k r| j | j
| qn | S( s Run the content of the origParagraph through preprocess, make sentences
Return a new paragraph containing the marked up sentences
R s .//text()R5 s
s utf-8s «»"t .t ?t !i ( R R[ t xpathR R| t Falset splitR\ t decodeR7 RU t len(
R t
origParagrapht newParagrapht allTextRu Rz t sentencet insideQuotet previousWordt word( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRP s2 0-
c C sJ t j d } t | j | j d /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRU + s
( R R! R" R RL Rh R| RP RU ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR2 s
(t Parallelizec B sz e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
d Z d
Z d Z
d Z RS(
s*
A class to parallelize two files
Input is the xml file that should be parallellized and the language that it
should be parallellized with.
The language of the input file is found in the metadata of the input file.
The other file is found via the metadata in the input file
c C s g | _ t t j j | | } | j j | t | j d j | j d j } | j j | | j r | j n d S( s
Set the original file name, the lang of the original file and the
language that it should parallellized with.
Parse the original file to get the access to metadata
i N(
t origfilesR R R
t abspathR\ R R t isTranslatedFromLang2t reshuffleFiles( R t origfile1t lang2t tmpfile( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR ? s ) c C s2 | j d } | j d | j d <| | j d /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR P s
c C s | j S( sE
Return the list of (the two) files that are aligned
( R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getFilelistX s c C s | j d j S( Ni ( R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getLang1^ s c C s | j d j S( Ni ( R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getLang2a s c C s | j d j S( Ni ( R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getOrigfile1d s c C s | j d j S( Ni ( R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getOrigfile2g s c C sk t } t j | j } | j } | j d } | d k rg | j d | j k rg t } qg n | S( sD
Find out if the given doc is translated from lang2
s .//translated_froms* {http://www.w3.org/XML/1998/namespace}langN(
R R R R R t findt NoneR R R7 ( R t resultR R t translated_from( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR j s c C s# t j j t j d d } t j j t j d d } t j d d | j d | j d t j d | | g d t j d
t j } | j \ } } | j
d k r t j d It
j Id
IJt j | IJt j | IJ| j
Sd | j | j d } t j j t j d | S( sb
Generate an anchor file with lang1 and lang2. Return the path to the anchor file
Ri s gt/common/src/anchor.txts gt/common/src/anchor-admin.txts generate-anchor-list.pls --lang1=s --lang2s --outdir=R$ Rl Rm i s Could not generate s into sentencess anchor-s .txt( R R
R R& Rn Ro R R Rp Rq Rr Rs Rm R' R ( R t infile1t infile2Ry Rz R{ t outFilename( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt generateAnchorFilex s R
c C s x | j D] } t j j | j } t j j | rx | j | } t | | j } | j | j
| q
t j | Id IJd Sq
Wd S( s`
Call corpus-analyse.pl which reads an xml file and makes it palatable for tca2
s
doesn't existi i (
R R R
R R t existst getSentFilenameR2 R RL Rh Rs Rm ( R R' t infileRe t divider( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt dividePIntoSentences s
c C s9 | j j d d } t j d d | | j d S( sr
Compute a name for the corpus-analyze output and tca2 input file
Input is a ParallelFile
s .xmlR5 R$ s /tmp/s _sent.xml( R R R R& R ( R R' t origfilename( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C s | j } t j d | | j | j d | j | j d g d t j d t j } | j \ } } | j d k r t j d I| j | j d Id I| j | j d Id IJt j | IJt j | IJn | j S( s2
Parallelize two files using tca2
s tca2.shi i Rl Rm s Could not parallelizet ands into sentences(
R Rn Ro R R Rp Rq Rr Rs Rm ( R t
anchorNameRy Rz R{ ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt parallelizeFiles s SK
( R R! R" R R R R R R R R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR 6 s t TestParallelizec B s_ e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
d Z RS(
s0
A test class for the Parallelize class
c C s! t t j d d d | _ d S( NR$ sA /prestable/converted/sme/facta/skuvlahistorja2/aarseth2-s.htm.xmlR% ( R R R& t parallelize( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR( s c C s( | j | j j t j d d d S( NR$ sA /prestable/converted/nob/facta/skuvlahistorja2/aarseth2-n.htm.xml( R) R R R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testOrigPath s c C s( | j | j j t j d d d S( NR$ sA /prestable/converted/sme/facta/skuvlahistorja2/aarseth2-s.htm.xml( R) R R R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testParallelPath s c C s | j | j j d d S( NR% ( R) R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testLang1 s c C s | j | j j d d S( NR- ( R) R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testLang2 s c C s8 | j | j j | j j d t j d d d S( Ni R$ s /tmp/aarseth2-n.htmnob_sent.xml( R) R R R R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGetSentFilename s c C s | j | j j d d S( Ni ( R) R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testDividePIntoSentences s c C s | j | j j d d S( Ni ( R) R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testParallizeFiles s c C s3 | j | j j t j j t j d d d S( NR$ s anchor-nobsme.txt( R) R R R R
R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGenerateAnchorFile s ( R R! R" R( R R R R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s t Tmxc B s_ e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
d Z RS(
s|
A class that reads a tmx file, and implements a bare minimum of functionality
to be able to compare two tmx'es
c C s
| | _ d S( N( t tmx( R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C s | j j d j d d S( s9
Get the srclang from the header element
s .//headert srclangi ( R R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt
getSrcLang s c C s | j S( s)
Get the tmx xml element
( R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getTmx s c C s d } y | | d d j j } Wn t k
r8 n X| d 7} y | | d d j j } Wn t k
ru n X| d 7} | j d S( s9
Extract the two strings of a tu element
R5 i s i s
s utf-8( RV t stript AttributeErrorR= ( R t tut string( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt
tuToString s
c C s> d } y | d j j } Wn t k
r0 n X| j d S( s9
Extract the string from the tuv element
R5 i s utf-8( RV R R R= ( R t tuvR ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt tuvToString s
c C s[ | j j d | d d i d d 6} g } x$ | D] } | j | j | q7 W| S( sm
Get all the strings in the tmx in language lang, insert them
into the list strings
s .//tuv[@xml:lang="s "]t
namespacess$ http://www.w3.org/XML/1998/namespacet xml( R R R\ R ( R R t all_tuvt stringsR ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt langToStringlist s
c C sF | j j d } g } x$ | D] } | j | j | q" W| S( sH
Extract all string pairs in a tmx to a list of strings
s .//tu( R R R\ R ( R t all_tuR R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt tmxToStringlist s
c C s y, | d d j j } | | d d _ Wn t k
r? n Xy, | d d j j } | | d d _ Wn t k
r n X| S( s'
Input is a tu-element
i i ( RV R R ( R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt prettifySegs" s
c C s | j j d } t j d } x[ | D]S } t j d } | j | d | j | d | j | } | j | q+ Wt j d } | j | | | _ d S( sL
Reverse the langs in a tmx
Return the reverted tmx
s .//tuRZ R i i R N( R R R R[ R\ R R ( R R RZ R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt reverseLangs4 s
( R R! R" R R R R R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s t TestTmxc B sV e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
RS( s(
A test class for the Tmx class
c C s t t j d | _ d S( Ns# parallelize_data/aarseth2-n.htm.tmx( R R R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR( M s c C s t j | d t } t j | d t } t j } | j | | d s | j t j d | | d j d } t
| n d S( s5
Check if two xml snippets are equal
R4 i R5 s utf-8N( R R6 R7 R R8 R9 R: R; R< R= R> ( R R? R@ RA RB RC RD ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRE P s *c C s | j | j j d d S( s$ Test the getSrcLang routine
R% N( R) R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGetSrcLang] s c C s/ t j d } | j | j j | d d S( Nsd SámegiellaSamisks Sámegiella Samisk
( R RO R) R R ( R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testTuToStringb s c C s/ t j d } | j | j j | d d S( Ns0 Sámegiellas Sámegiella( R RO R) R R ( R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testTuvToStringg s c C s t d d } | j } g } g } xE | D]= } | j d } | j | d | j | d j q. W| j | j j d | | j | j j d | d S( Ns* parallelize_data/aarseth2-n.htm.tmx.as.txtt rs i i R% R- ( Ra t readlinesR R\ R R) R R ( R Rf t
stringListt nobListt smeListR t pairList( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testLangToStringListl s
c C sB t d d } | j } | j | j | j j | d S( Ns* parallelize_data/aarseth2-n.htm.tmx.as.txtR ( Ra R Rd R) R R ( R Rf t wantList( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testTmxToStringlistz s
c C s> t j d } t j d } | j | j j | | d S( Ns ubba gubba. ibba gibba.abba gabba. ebba gebba.s ubba gubba. ibba gibba.
abba gabba. ebba gebba.
( R RO RE R R ( R t wantXmlt gotXml( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testPrettifySegs s ( R R! R" R( RE R R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR I s
t TmxFromTca2c B sh e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
d Z d
Z RS( sA
A class to make tmx files based on the output from tca2
c C s# | | _ t j | | j d S( s9
Input is a list of ParallelFile objects
N( t filelistR R t setTmx( R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C s_ t j d } | j | j | | j d j | j | j | | j d j | S( sI
Make a tmx tu element based on line1 and line2 as input
R i i ( R R[ R\ t makeTuvR R ( R t line1t line2R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt makeTu s &&c C s] t j d } | | j d /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s
!
c C sT t j d } d | j d
Make a tmx header based on the lang variable
t headerR t segtypes
OmegaT TMXs o-tmfs en-USt adminlangR t plaintextt datatype( R R[ R ( R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt
makeTmxHeader s
c C s7 | j d d } t j d } | j d | } | S( s7
Remove the s tags that tca2 has added
s R5 s ( R t ret compilet sub( R R t sregex( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C s d | j d j d } d | j d j d | j d j d } | j d j j | | } | j d j j d d } t j j | | S( s2
Compute the name of the tmx file
s /converted/i t /s /tmx/t 2i s .xmls .tmx( R R R R R R R
R ( R t origPathPartt replacePathPartt
outDirnameR ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getOutfileName s
0 c C sz | j } yQ t | d } t j | j } | j | d t d d d t | j Wn t k
ru d GHn X| S( s<
Write a tmx file given a tmx etree element
R^ R4 R_ s utf-8R` s ouch, printTmxFile( R Ra R Rb R Rc R7 Rd t IOError( R R Rf Rg ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt printTmxFile s
c C s t j d } | j | j d j } | j | | j | j d } | j | j d } t j | d } x? t d | | D]+ \ } } | j
| | } | j | q W| S( sE
Make tmx file based on the two output files of tca2
R i i RZ N( R R[ R R R R\ t readTca2Outputt
SubElementt mapR R ( R R R t pfile1_datat pfile2_dataRZ R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s
c C s{ d } | j | j d d } y) t | d } | j } | j Wn* t k
rv \ } } d j | | GHn X| S( sI
Read the output of tca2
Input is a ParallelFile
R5 s .xmls _new.txtR s I/O error({0}): {1}( R R Ra R Rd R t format( R R' RV t pfileNameRf t errnot strerror( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C s9 | j j d d } t j d d | | j d S( sr
Compute a name for the corpus-analyze output and tca2 input file
Input is a ParallelFile
s .xmlR5 R$ s /tmp/s _sent.xml( R R R R& R ( R R' R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s (
R R! R" R R R R R R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s
t TestTmxFromTca2c B sV e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
RS( s0
A test class for the TmxFromTca2 class
c C s3 t t j d d d } t | j | _ d S( sK
Hand the data from the Parallelize class to the tmx class
R$ sA /prestable/converted/sme/facta/skuvlahistorja2/aarseth2-s.htm.xmlR% N( R R R& R R R ( R t para( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR( s c C s t j | d t } t j | d t } t j } | j | | d s | j t j d | | d j d } t
| n d S( s5
Check if two xml snippets are equal
R4 i R5 s utf-8N( R R6 R7 R R8 R9 R: R; R< R= R> ( R R? R@ RA RB RC RD ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRE s *c C sD d } d } | j j | | } t j d } | j | | d S( Ns3 ubba gubba. ibba gibba.s3 abba gabba. ebba gebba.s ubba gubba. ibba gibba.abba gabba. ebba gebba.( R R R RO RE ( R R R t gotTut wantTu( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt
testMakeTu% s
c C sD d } d } | j j | | } t j d } | j | | d S( Ns3 ubba gubba. ibba gibba.t smis< ubba gubba. ibba gibba.( R R R RO RE ( R R R t gotTuvt wantTuv( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testMakeTuv/ s
c C s; d } | j j | } t j d } | j | | d S( NR sd ( R R R RO RE ( R R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testMakeTmxHeader8 s c C s, | j j d } d } | j | | d S( Ns3 ubba gubba. ibba gibba.s ubba gubba. ibba gibba.( R R R) ( R R? R@ ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testRemoveSTag@ s c C s3 | j | j j t j j t j d d d S( NR$ s> prestable/tmx/nob2sme/facta/skuvlahistorja2/aarseth2-n.htm.tmx( R) R R R R
R R& ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGetOutfileNameF s c C sH t j d } | j j t j | j j } | j | | d S( Ns# parallelize_data/aarseth2-n.htm.tmx( R R R R R RE ( R R@ R? ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testPrintTmxFileI s
( R R! R" R( RE R R R
R R R
( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s
t
TmxComparatorc B s; e Z d Z d Z d Z d Z d Z d Z RS( s*
A class to compare two tmx-files
c C s | | _ | | _ d S( N( t wantTmxt gotTmx( R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR T s c C s t | j j S( sA
Return the number of lines in the reference doc
( R R R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getLinesInWantedfileY s c C s\ d } xO t j | j j | j j d d D]# } | d d k r1 | d 7} q1 q1 W| S( s
Given a unified_diff, find out how many lines in the reference doc
differs from the doc to be tested. A return value of -1 means that
the docs are equal
it ni i t -( t difflibt unified_diffR R R ( R t numDiffLinesR ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getNumberOfDifferingLines` s
1c C sL g } x? t j | j j | j j d d D] } | j | q1 W| S( s?
Return a stringlist containing the diff lines
R i ( R R R R R R\ ( R t diffR ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt
getDiffAsTextn s 1c C sV g } xI t j | j j | | j j | d d D] } | j | d q7 W| S( s?
Return a stringlist containing the diff lines
R i s
( R R R R R R\ ( R R R R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getLangDiffAsTextx s 7( R R! R" R R R R R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR P s
t TestTmxComparatorc B s e Z d Z d Z RS( s2
A test class for the TmxComparator class
c C sy t t t j d t t j d } | j | j d | j | j d | j t | j d d S( Ns# parallelize_data/aarseth2-n.htm.tmxii
i ( R R R R R) R R R R ( R t comp( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testEqualTmxes s -( R R! R" R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s t TmxTestDataWriterc B sV e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
RS( s5
A class that writes tmx test data to a file
c C sZ | | _ y& t j | } | j | j Wn$ t k
rU | j | j n Xd S( N( t filenameR R t setParagsTestingElementR R t makeParagstestingElement( R R t tree( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s
c C s | j S( N( R ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getFilename s c C s: t j d } | | j d <| | j d <| | j d <| S( s;
Make the element file, set the attributes
t fileR t gspairst diffpairs( R R[ R ( R R R% R& t fileElement( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt makeFileElement s
c C s
| | _ d S( N( t
paragstesting( R R) ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s c C s t j d } | | j d <| S( s=
Make the testrun element, set the attribute
t testrunt datetime( R R[ R ( R R+ t testrunElement( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt makeTestrunElement s
c C s t j d } | S( s0
Make the paragstesting element
R) ( R R[ ( R t paragstestingElement( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR! s c C s | j j d | d S( Ni ( R) t insert( R R* ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt insertTestrunElement s c C sn yQ t | j d } t j | j } | j | d t d d d t | j Wn t k
ri d GHn Xd S( s8
Write the paragstesting data to a file
R^ R4 R_ s utf-8R` s ouch, ParagstestingresultsN( Ra R R Rb R) Rc R7 Rd R ( R Rf Rg ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt writeParagstestingData s
( R R! R" R R# R( R R- R! R0 R1 ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s t TestTmxTestDataWriterc B sV e Z d Z d Z d Z d Z d Z d Z d Z d Z d Z
RS( s+
A class to test TmxTestDataWriter
c C s t d | _ d S( Nt testfilename( R t writer( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR( s c C s t j | d t } t j | d t } t j } | j | | d s | j t j d | | d j d } t
| n d S( s5
Check if two xml snippets are equal
R4 i R5 s utf-8N( R R6 R7 R R8 R9 R: R; R< R= R> ( R R? R@ RA RB RC RD ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRE s *c C s | j | j j d d S( NR3 ( R) R4 R# ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testGetFilename s c C s; t j d } | j j d d d } | j | | d S( Ns/ t abct 634t 84( R RO R4 R( RE ( R t wantElementt
gotElement( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testMakeFileElement s c C sZ t j d } | j j d } | j j d d d } | j | | j | | d S( Ns[ s
20111208-1234R6 R7 R8 ( R RO R4 R- R( R\ RE ( R R9 R: R' ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testMakeTestrunElement s
c C sv t j d } | j j } | j j d } | j j d d d } | j | | j | | j | | d S( Nsz s
20111208-1234R6 R7 R8 ( R RO R4 R! R- R( R\ RE ( R R9 R: R, R' ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testMakeParagstestingElement s
c C s t j d } | j j } | j j | | j j d } | j j d d d } | j | | j | | j j d } | j j d d d } | j | | j j | | j | | d S( Ns s
20111208-1234R6 R7 R8 s
20111208-2345(
R RO R4 R! R R- R( R\ R0 RE ( R R9 R: R, R' ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testInsertTestrunElement s
c C s t j d } | j j } | j j | | j j d } | j j d d d } | j | | j | | j j t j | j j
} | j | | d S( Nsz s
20111208-1234R6 R7 R8 ( R RO R4 R! R R- R( R\ R1 R R RE ( R R@ R: R, R' R? ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt testWriteParagstestingData$ s
( R R! R" R( RE R5 R; R< R= R> R? ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR2 s
t TmxGoldstandardTesterc B sb e Z d Z d
d Z d Z d Z d Z d Z d Z d Z
d Z d Z RS( sL
A class to test the alignment pipeline againt the tmx goldstandard
c C sM d | _ t | | _ | d k r6 | j | _ n | j | | _ d S( sn
Set the name where the testresults should be written
Find all goldstandard tmx files
i N( t numberOfDiffLinesR t testresultWriterR t
dateformatt date( R t testresult_filenamet dateformat_addition( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR 8 s
c C s | j | 7_ d S( sI
Increase the total number of difflines in this test run
N( RA ( R t diffLines( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt setNumberOfDiffLinesD s c C s | j S( N( RA ( R ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt getNumberOfDiffLinesJ s c C s= d d l } d d l } | j j | j } | j d S( sQ
Get the date and time, 20111209-1234. Used in a testrun element
iNs %Y%m%d-%H%M( R+ t timet
fromtimestampt strftime( R R+ RJ t d( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRC M s c C s | j j | j } d } xX | j D]J } d G| Gd GH| j d d k rY d } n d } | j | | | q( W| j j | | j j d S( NR5 t testings ...t nob2smeiR- R% ( RB R- RD t findGoldstandardTmxFilesR t
alignFilesR0 R1 ( R R* R t wantTmxFile( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt runTestW s
c C s | j | } t | | } | j d k r | j d k r | j } t | } t t j | } t | | } | j
j | d j t
| j t
| j }
| j | j | j |
| j | | | d j q n d S( s
Align files
Compare the tmx's of the result of this parallellization and the tmx of the goldstandard file
Write the result to a file
Write the diffs of these to tmx's to a separate file
i N( t computeXmlfilenameR R R R R R R R R RB R( R R R R RH R\ t writeDiffFiles( R R* RR R t xmlFilet parallelizerR R R t
comparatorR' ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRQ n s :
c C sL | j d d } | j d d } | j d d } | j d d } | S( sI
Compute the name of the xmlfile which should be aligned
s tmx/goldstandard/s
converted/RO R% t sme2nobR- s .tmxs .xml( R ( R RR RV ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRT s
c C sa d G| GH| d | j d } t j j t j j | j j d } y" t t j j | | d } Wn6 t k
r d Gt j j | | GHt j
d n X| j d | d | j d
| j | j
| j d | j d | j | j | j | j d | j d | j | j | j | j d
| j d S( s/
Write diffs to a jspwiki file
RU t _s .jspwikit tca2testingR^ s couldn't write filei s !!!s
s !!TMX diff
{{{
s
}}}
!!s
diff
{{{
s
}}}
N( RD R R
R R RB R# Ra R Rs Rt Rc t
writelinesR R R R Rd ( R RX RW R R Rf ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRU s" *"
c C s t j d t j j t j d d d d d g d t j d t j } | j \ } } | j d k r t j
d
IJt j
| IJt j d n | j d } | d
Sd S( sH
Find the goldstandard tmx files, return them as a list
R R$ s prestable/tmx/goldstandards -names *.tmxs -printRl Rm i s* Error when searching for goldstandard docsi s
iN(
Rn Ro R R
R R& Rp Rq Rr Rs Rm Rt R ( R Ry Rz R{ t files( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyRP s F
N(
R R! R" R R RH RI RC RS RQ RT RU RP ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR@ 4 s
# t TmxFixerc B s e Z d Z d Z RS( s
A class to reverse the langs and change the name of a tmx file if needed
Possible errors of a tmx file:
* the languages can be in the wrong order
* the name is wrong
* the file is placed in the wrong lang directory
c C s d S( sA
Input is the tmx file we should consider to fix
N( ( R t filetofix( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR s ( R R! R" R ( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyR^ s t __main__( ( ( ( ( ( ( ( (# R R Rs Rn R R; t lxmlR R t unittestR t TestCaseR# R1 R2 R R R R R R R R R R2 R@ R^ R t testt TestSuitet testSuitet addTestt makeSuitet TextTestRunnert run( ( ( s> /home/boerre/langtech/trunk/gt/script/langTools/parallelize.pyt s: >Sj u=D2!AP "