\documentclass{llncs}
\usepackage{makeidx}  % allows for indexgeneration
%
\begin{document}
\pagestyle{empty}
%
\mainmatter
%
\title{Evaluation of Tectogrammatical\\ Annotation of PDT}
%First Observations}
\subtitle{First Observations}
\author{Eva Haji\v{c}ov\'a \and Petr Pajas}
\institute{Faculty of Mathematics and Physics, Charles University,
Prague, Czech Republic\\ \email{\{hajicova,
pajas\}@ufal.ms.mff.cuni.cz}} \maketitle
%
\begin{abstract}
Two phases of an evaluation of annotating a Czech text corpus on
an~underlying syntactic level are described and the results are compared
and analyzed.
\end{abstract}
%
\section{Introduction: Tectogrammatical annotation of the Prague Dependency Treebank}
%
Tagging of the Prague Dependency Treebank (PDT in the sequel) on
the underlying syntactic layer (resulting in tectogrammatical
tree structures, TGTSs) is an ambitious task the realization of
which must be carefully supervised and regularly evaluated, in
order to reach the proclaimed aims and to obtain (theoretically
and applicationally) interesting and applicable results. In the
present contribution we describe one of the steps in the
introduction of large-scale tagging of a very large corpus.

Tectogrammatical tagging of PDT is carried out in two phases:
\\
(i) an automatic preprocessing transforming the analytic tree
structures (ATSs) into structures that are half-way to the TGTSs,
\\
(ii) manual ``shaping'' of the TGTSs into their final forms. The
human annotators have at their disposal a manual \cite{eh99} and
there are regular (weekly) instructive sessions.
%
\section{Description of the evaluation experiment}
%
In order to evaluate the quality of the manual and the
instructive sessions and to make estimates about the difficulty
of the tagging task (as well as to predict the speed of tagging)
we have carried out the following experiment:

Three annotators (all linguists with a university-level
education, two having a PhD in linguistics) were given one
(randomly chosen) sample of (newspaper) text taken from the Czech
National Corpus, which consisted of 50 sentences, with their ATSs
preprocessed by the automatic preprocessing procedure mentioned
above. They had the manual at their disposal and were asked to
tag the sentences according to the manual without negotiations
among themselves about the unclear issues. The task of the
annotators was to check the dependency structure as such and to
assign to the particular values of the dependency relations
(functors). They also were supposed to check the lexical values
of the nodes and to add appropriate lexical values in case they
added some node in the TGTS that was deleted in the surface
structure and therefore was missing also in the ATS. A special
programme was written to compare the results of the three
annotators sentence per sentence (actually, word by word (or node
by node)) and to summarize some statistical and qualitative
results.

After the evaluation of the results and after a thorough
discussion during several instructive sessions about the points
in which the annotators differed, we have repeated the same task
with the same annotators annotating another randomly chosen set
of 47 sentences, to compare the results in order to obtain some
judgements about the degree of improvements of the quality of
tagging and also to make some predictions about the
speed. 
%
\section{The first round of the experiment}
\label{sect:tri}
%
Out of the total of 50 sentences, only for 10 of them all the
annotators gave the same TGTSs; since the number of occurrences
of dependency relations in these sentences was not greater than
3, this is a negligible portion of the whole set. The
distribution of the number of sentences and the number of
differences (one difference means that one dependency relation or
a part of a label of a node was assigned in a different way by
one of the annotators) is displayed in Table~\ref{table:tab1}.
\setlength{\tabcolsep}{4pt}%
\begin{table}%
\begin{center}%
\caption{The distribution of the number of sentences and the
number of differences} \label{table:tab1}%
\begin{tabular}{llllllllllllllll}%
\hline \noalign{\smallskip}%
No. of diff. & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 9 & 10 & 11 & 13 & 14 & 20 & 21 & 27\\
%\noalign{\smallskip}
\hline
\noalign{\smallskip}%
No. of sent. & 3 & 4 & 4 & 5 & 3 & 3 & 4 & 1 & 6 & 2 & 1 & 1 & 1 & 1 & 1\\
\hline
\end{tabular}%
\end{center}%
\end{table}%
\begin{table}%
\begin{center}%
%\vskip6mm%
\caption{Distribution of sentences according to the number of
differences in each sentence ignoring the differences in lemmas} \label{table:tab1x}%
\begin{tabular}{lllllllllllllll}%
\hline \noalign{\smallskip}%
No. of diff. & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 18 & 20 & 21\\
%\noalign{\smallskip}
\hline \noalign{\smallskip}%
No. of sent. & 6 & 4 & 3 & 7 & 4 & 3 & 3 & 2 & 2 & 2 & 1 & 1 & 1 & 1\\
\hline
\end{tabular}%
\end{center}%
\end{table}%
\setlength{\tabcolsep}{1.4pt}%
Let us note that the number of dependency relations is slightly
smaller than the number of words in the sentence, due to the fact that
more nodes get taken away than newly restored.
The total number of occurrences of
dependency relations (i.e. edges) in the test set was 720; the
number of differences was 290. 
%%%%%%%%%%%%% New
Out of this total number of
differences, there were 56 differences in lemmas,
64 differences in the determination
of which node depends on which node, and 58 differences in the
restoration of nodes not appearing in the ATSs but restored - at
least by one of the annotator - in the TGTSs. 
This leads us to the total of 178 differences in other features than the values of
the dependency relations, i.e. out of the total of 720
occurrences of dependency relations (edges) in the first set of
sentences there were 112 differences in the assignment of the
values of these relations, and 64 differences in the
establishment of the edges.
%%%%%%%%%%%%%
It is interesting to note that
while the trees with difference 0 were more than simple, the
trees with the number of differences $N=1$ were rather complicated
(including 13, 9, 18 relations, respectively), and the same holds
about $N=2$ ($12, 5, 9, 15$), and $N=3$ ($26,11,17,4$). The sentences
with $N>10$ were almost all complex sentences with one
or more coordinated structures ($N=11,13,20$), with several general
participants expressed by a zero morph ($N=21$), structures with
focus-sensitive particles in combination with
negation and with general participants ($N=14$), and a structure
with several surface deletions that should be restored in the
TGTS ($N=27$).

The following observations seem to be important:

(a) The dependency relations (i.e. edges) were correctly
established in all cases with the following exceptions:
\\
(i) the position of focussing particles.
\\
(ii) the apposition relation was attached differently in one case
\\
(iii) in several cases, the edges for obligatory though
(superficially) deletable complementations of verbs were missing
with one or two annotators.

Improvements to be done: for (i) and (ii), more specific
instructions should be formulated in the manual; (iii) will
improve once the basic lexicon is completed with the assignment
of verbal frames specifying the kinds of obligatory
complementations.

(b) Lexical labels: the differences concerned uncertainties in
assigning the value Gen (for a general participant) and `on'
(pronoun `he' used in pro-drop cases) and cor (used in cases of
control). Improvement: these cases are well-definable and should
be more clearly formulated in the manual.

(c) Values of dependency relations: The instructions give a
possibility to put a question mark if the annotator is not sure
or to use alternatives (two functors). The differences mostly
concern uncertainties of the annotators when they try to decide
in favour of a single value; other differences are rather rare
and concern issues that are matters of linguistic discussions.

%
\section{The second round of the experiment}
\label{sect:ctyri}
%
In the second round of the task, we have evaluated the assignment
of TGTSs to 47 sentences in another randomly chosen piece of text
(again, taken from the newspaper corpus). When analyzing the
results, we faced a striking fact (not so prominent in the first
round): there was a considerable amount of differences in the
shape rather than in the value of the lexical tags, esp. with
lemmas of the general participants (Gen vs. gen) and of the added
nodes for coreferring elements (Cor vs. cor). Also other
differences in the lemmas were rather negligible caused just by
certain changes in the instructions for the annotators.

In Table~\ref{table:tab2}, we count all differences and in
Table~\ref{table:tab3} we ignore the differences in lemmas. A
comparison of the two Tables e.g. shows that if differences in
lemmas are ignored, the number of sentences with the number of
differences equal to 0 through 2 increases from 20 to 26, and
that the number of sentences with the number of differences
greater than 7 decreases from 10 to 5.
\setlength{\tabcolsep}{4pt}%
\begin{table}%
\begin{center}%
\caption{Distribution of sentences according to the total number
of differences in each sentence} \label{table:tab2}%
\begin{tabular}{llllllllllllllllll}%
\hline \noalign{\smallskip}%
No. of diff. & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 14 & 16 & 18 & 29\\
%\noalign{\smallskip}
\hline \noalign{\smallskip}%
No. of sent. & 5 & 8 & 7 & 4 & 4 & 5 & 2 & 2 & 1 & 1 & 1 & 2 & 1 & 1 & 1 & 1 & 1\\
\hline
 \end{tabular}%
 \end{center}%
 \end{table}%
 \begin{table}%
 \begin{center}%
%\vskip6mm%
\caption{Distribution of sentences according to the number of
differences in each sentence ignoring the differences in lemmas} \label{table:tab3}%
\begin{tabular}{lllllllllllll}%
\hline \noalign{\smallskip}%
No. of diff. & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 11 & 12 & 14 & 28\\
%\noalign{\smallskip}
\hline \noalign{\smallskip}%
No. of sent. & 8 & 7 & 11 & 2 & 6 & 4 & 2 & 2 & 2 & 1 & 1 & 1\\
\hline
\end{tabular}%
\end{center}%
\end{table}%
\setlength{\tabcolsep}{1.4pt}%
The total number of occurrences of
dependency relations (i.e. edges) in the second test set was 519;
the number of differences was 239. Out of this total number of
differences, there were 54 differences in lemmas, 1 difference in
the assignment of modality, 35 differences in the determination
of which node depends on which node, and 43 differences in the
restoration of nodes not appearing in the ATSs but restored - at
least by one of the annotator - in the TGTSs. This leads us to
the total of 133 differences in other features than the values of
the dependency relations, i.e. out of the total of 519
occurrences of dependency relations (edges) in the second set of
sentences there were 106 differences in the functors.

In contrast to the first round of the experiment, in the second round
the trees with differences 0 were comparatively rich in the number of
relations they contained; having 2, 2, 4, 9, and 10 relations if all
differences are taken into account, and if the differences in lemmas
are ignored, the set of sentences without differences is even enriched
by sentences with 7, 11, and 17 relations. The trees with the number
of differences $N=1$ were again rather complicated (including 4, 6, 7,
9, 9, 10, 11, 17 relations, if all differences are taken into
account), and the same holds about $N=2$ ($5, 5, 7, 7, 8, 11, 15$),
and $N=3$ ($8, 8, 11, 13$). Similarly as in the first round, the
sentences with $N>10$ included differences in the assignment of
general participants expressed by zero morphs (this is true about all
sentences in this group), and in most of them the same differences
were repeated because of the fact that the sentences included
coordination.
%
\section{Comparison}
\label{sect:pet}
%
To make
Tables \ref{table:tab1} and \ref{table:tab2}.
comparable, we exclude
the number of
sentences with $N=0$: this group was of no importance in the first
round because the sentences included there were very poor in the
number of relations they contained, but in the second round this
figure is rather important because the sentences belonging there
are rather complex (having 2, 2, 4, 9, and 10 relations if all
differences are taken into account, and if the differences in
lemmas are excluded, this set of is even enriched by sentences
with 7, 11, and 17 relations).
% \setlength{\tabcolsep}{4pt}%
% \begin{table}%
% \begin{center}%
% \caption{Distribution of sentences according to the total number
% of differences in each sentence; the number of sentences with $N=0$
% is excluded} \label{table:tab4}%
% \begin{tabular}{lllllllllllllllll}%
% \hline \noalign{\smallskip}%
% No. of diff. & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 14 & 16 & 18 & 29\\
% %\noalign{\smallskip}
% \hline \noalign{\smallskip}%
% No. of sent. & 8 & 7 & 4 & 4 & 5 & 2 & 2 & 1 & 1 & 1 & 2 & 1 & 1 & 1 & 1 & 1\\
% \hline
% \end{tabular}%
% \end{center}%
% \end{table}%
% \setlength{\tabcolsep}{1.4pt}%
In total, there are 40 sentences
taken into account in Table \ref{table:tab1} and 42 in Table
% \ref{table:tab4}; 
\ref{table:tab2};
out of this total number, 19 sentences in the
first round contain less than 5 differences, the rest includes
more differences; this number improves in the second round, in
which there are 28 with less than 5 differences, i.e.
an improvement of almost 50\%.

(i) There was a considerable improvement in the assignment of the
values of the dependency relations if compared with the first
round: out of the total of 123 differences, 21 are not real
differences because they consist in an assignment of a ``double
functor'' (or a ``slashed'' value) by some of the annotators and
only one of the values of such a double functor by the other(s).
The possibility of an assignment of two (or even more)
alternatives to a simple node was introduced in order to
make it possible for an annotator to express his/her uncertainty
in case even the context does not make it clear what particular
relation is concerned (eg. ACMP/COND - Accompaniment or
Condition; EFF/RESL - Effect of Result; AIM/BEN - Aim or
Benefactive). The introduction of the slashed values is very
important for the future research in the taxonomy of dependency
relations (merging two current types into one, or making more
distinctions) based on the corpus, or formulating the criteria
for the distinction between particular values in a more explicit
way. In any case, however, the agreement between the annotators
on one of the values (and the disregard of the other value by
other annotators) should not be really counted as a difference.

There remain, of course, differences which have to be reduced in
the further course of the annotation task. The following
observations seem to be important for the future development:

(i) As already noticed in (a)(iii) in Sect. \ref{sect:tri} above,
in several cases, the edges for obligatory though (superficially)
deletable complementations of verbs or nouns were missing with
one or two annotators. There has been a considerable improvement
over the first round since the instructions in the manual have
been made more precise in that the restoration of deletable
complementations of nouns is restricted to deverbatives in the
strict sense, specified by productive derivational means (endings
such as {\it -\'an\'{\i}, -en\'{\i}}). However, there still were cases
where the annotators added nodes in cases which were excluded by
the instructions ({\it prodejce} `seller', {\it rozhodnut\'{\i}} `decision')
or were not certain if they are supposed to distinguish two
meanings of the deverbative ({\it uzn\'an\'{\i}} `recognition' or
`recognizing', {\it pl\'anov\'an\'{\i}} `planning' or `the result of
planning'). This is really a difficult point and we may only hope
that a better routine will be acquired by the annotators during
the annotation process.

(ii) Another case of incorrect restoration is connected with the
different types of `reflexive' forms in Czech. In the TGTSs, a
distinction should be made between cases where the reflexive form
of the verb is equivalent to a passive (the so-called reflexive
passive is very frequent especially in technical texts) , or
whether the particle `{\it se}' is an integral part of the verb (esp. the
so-called reflexivum tantum). Examples of the former type
occurring in our sample are the forms {\it \v{s}lo se} `they went',
{\it vytv\'a\v{r}\'{\i} se} `(it) is created'; in these case, the lemma
`{\it se}' of the corresponding node in the ATS is `rewritten' to the
lemma of a general participant (gen) and gets the functor of Act;
the subject of the (surface) construction gets the functor Pat. In
the latter type of reflexive verbs, the ATS node with the label
`{\it se}' is deleted and the particle `{\it se}' is added to the lexical
label of the verb; this is the case of verbs such as
{\it specializovat se} `specialize', {\it orientovat se} `orientate oneself',
{\it p\v{r}edstavit} si `imagine', {\it pustit se} `to get involved in',
{\it zab\'yvat se} `occupy oneself with'. Improvement should be reached
by more explicit (and more thoroughly exemplified) instructions
in the manual.

(iii) Another considerable improvement concerned the cases of
lexical labels assigning the value Gen (for a general
participant), `{\it on}' (pronoun `he' used in pro-drop cases) and
cor (used in cases of control). The trivial mistake in the outer
shape of the labels (Gen or gen, Cor or cor) will be
removed by a macro assigning these values automatically.

(iv) A similar unifying measure should be taken for cases of the
assignment of lemmas for pronouns (the lemma of a personal
pronoun should be assigned also in cases of a possessive use of
pronouns), for the assignment of lemmas to nodes representing
certain (meaningful) punctuation marks, and for adding `empty
verbs' in cases when this is necessary for an adequate acount of
the dependency structure of the sentence.

% (v) As for differences in the assignment of individual values of
% the dependency relations (of functors), the following variations
% have been found in the second round of the experiment (the
% capitals indicate the tag values):
% \\ ACTor x APPurtenance \\ AIM x
% INTention x PATient \\ APPurtenance x PATient \\ AIM x RESTRictive
% attribute \\ AIM x DIRectional where-to x INTention \\
% ACMP(Accompaniment) x RESTRictive attribute \\ ACTor x LOCation \\
% DIRectional where-to x EFFect \\ DIRectional where-to x PATient
% \\ IDentity x PATient x RESTRictive attribute \\
% MANNer x PREC (reference to a preceding sentence) \\ PATient x
% PREDicate \\ PATient x REGard \\ PATient x RESTRictive attribute \\
% PREC (reference to a preceding sentence) x Temporal WHEN \\
% PREDicate x RESuLt\\ temporal How Often x Temporal WHEN \\
% TOWHen x Temporal WHEN
% \\
% It may be interesting to note that though not instructed in such
% a way, the annotators have never used the possibility to assign
% `???' in case they were not sure about the value except for cases
% of slashed values (this happened 20 times, mostly in combination
% with the value of Restrictive attribute).

To conclude, we believe that the results of both rounds of the
evaluation and their comparison will help us to improve the
manual for the further process of annotation in order to give
better specifications and to help to speed up the work of the
annotators. We have also gained several stimuli for linguistic
research in areas that have not yet been adequately described in
any Czech grammar.
%
\subsubsection{Acknowledgements.}
We would like to thank our colleagues Jan Haji\v{c} for giving us
an impetus for the evaluation experiments, Alena
B\accent127ohmov\'a for providing technical help in carrying them
out, and Petr Sgall for an intensive and continuous collaboration
on writing and modifying the manual. - Research for this paper
was supported mainly by the grant of the Czech Ministry of
Education VS96151 and partly also by the grant of the Czech
Grant Agency GA\v{C}R 405/96/K214.
%
\begin{thebibliography}{1}
% \bibitem{jh}
% Haji\v{c}, J.: Building a syntactically annotated corpus: The
% Prague Dependency Treebank. In: Issues of Valency and Meaning.
% Studies in Honour of Jarmila Panevová (ed. by E. Haji\v{c}ov\'a),
% Prague: Karolinum (1987)106-132
% \bibitem{eh98}
% Haji\v{c}ov\'a, E.: Prague Dependency Treebank: From analytic to
% tectogrammatical annotations. In: Text, Speech, Dialogue (eds.
% Petr Sojka, V\'aclav Matou\v{s}ek, Karel Pala and Ivan
% Kope\v{c}ek), Brno: Masarykova univerzita (1998) 45-50
\bibitem{eh99}
Haji\v{c}ov\'a E., Panevov\'a J., Sgall P.: Manu\'al pro
tektogramatick\'e zna\v{c}kov\'an\'\i, \'UFAL Technical Report
TR-1999-07, Universitas Carolina Pgragensis (1999)
% \bibitem{hps}
% Haji\v{c}ov\'a, E., Panevov\'a, J., Sgall, P.: Language Resources
% Need Annotations To Make Them Really Reusable: The Prague
% Dependency Treebank. In: Proceedings of the First International
% Conference on Language Resources \& Evaluation, Granada, Spain
% (1998) 713-718
\end{thebibliography}
%
\end{document}
