\documentclass{llncs}

\usepackage[dvips]{graphicx}
\usepackage[latin1]{inputenc}
\input epsf

\title{An annotation scheme for dialogues applied to anaphora resolution algorithms}
\author{Patricio Mart\'{\i}nez-Barco and Manuel Palomar}

\institute{Departmento de Lenguajes y Sistemas Inform\'aticos\\
Universidad de Alicante \\ Carretera de San Vicente del Raspeig -
Alicante - Spain \\ Tel. +34965903653 Fax. +34965909326\\
\{patricio, mpalomar\}@dlsi.ua.es}
\begin{document}


\label{firstpage} \maketitle

\begin{abstract}

Usually, anaphora resolution researchers focus their works on
defining a set of sources to extract useful information that helps
to anaphora resolution. Other works base their research on
extracting relationships between utterances and anaphors obtaining
satisfactory results working on languages with short anaphoric
accessibility spaces like English. In this work we state that
anaphora resolution in Spanish needs an adequate definition of
accessibility spaces and then we propose an annotation scheme for
dialogue structure to define these spaces.  This proposal has been
tested achieving successful results when it is applied to real
man-to-man telephone conversations.

\end{abstract}



\section{Introduction}

 In this paper, an annotation scheme for dialogues applied to
anaphora resolution algorithms is presented. This annotation
scheme for Spanish dialogues is based on the theories about the
taking of speaking turns. This scheme is oriented to the
definition of an anaphoric accessibility space where the
antecedents of an anaphor can be found. The adequate definition of
this space allows to optimize the anaphora resolution algorithm.

According to Dahlb\"{a}ck \cite{Dahlback91}, the efforts made so
far towards resolving anaphora can be divided into two basic
approaches : Traditional and Discourse-oriented. The traditional
approach generally depends on linguistic knowledge. In the
discourse-oriented approach, however, the researcher tries to
model the complex structure of discourse. Anaphora, accepted as
discourse phenomena, is resolved with the help of that complex
structure. These works are mostly focused on defining anaphora
resolution algorithms, both the traditional approaches (Hobbs
\cite{Hobbs86}, Baldwin \cite{Baldwin97}, Mitkov \cite{Mitkov98})
and the discourse-oriented ones (Grosz et al \cite{Grosz95},
Strube \cite{Strube99}).

However, the former do not perform a defined proposal about
anaphoric accessibility space, and the latter constraint the space
for possible antecedents to the previous utterance. Although, this
strategy is adequate for English processing, its application to
other languages such as Spanish is not such suitable. For
instance, Spanish personal pronouns contain more morphological
information than English ones. This makes Spanish speakers to
expect larger anaphoric accessibility spaces. Also, Spanish
free-word-order causes problems when grammatical role information
must be used.

Following the idea that any treatment of anaphora must consider
the structure of its source text-types, Fox \cite{Fox87}
investigated anaphora looking for an adequate accessibility space
on the basis of the relationship between discourse structure and
this phenomena. In this work, Fox used the theories of
conversational analysis by Sacks, Schegloff and Jefferson
\cite{Sacks74} to define the structure of discourse. According to
this approach, the adjacency pair is the basic structural unit of
dialogues. Thus, a close relationship between adjacency pairs and
anaphoric accessibility space is shown.


In the next section an annotation scheme for Spanish dialogue
structure is proposed. After that a brief survey of the anaphora
resolution algorithm is shown, and finally, the evaluation of the
annotation scheme applied to the anaphora resolution algorithm is
presented.

\section{An annotation scheme for dialogue structure}

For the successful processing and resolution of anaphora in
dialogues, we believe that the proper annotation of the dialogue
structure is necessary. With such a view, we propose an annotation
scheme, for Spanish dialogues, that is based on the work carried
out by Gallardo \cite{Gallardo96}, who applies, to Spanish
dialogues, the theories put forward by Sacks, Schegloff and
Jefferson \cite{Sacks74} about the taking of speaking turns
(conversational). According to these theories, the basic unit of
knowledge is the {\it move} that can inform the listener about an
action, request, question, etc. These moves are carried out by
means of {\it utterances}. Therefore, utterances are joined
together to become {\it turns}.

Since our work was done on spoken dialogues that have been written
(transcribed), the turn appears annotated in the texts and the
utterances are delimited by the use of punctuation marks. The
reading of a punctuation mark (., ?, !, ...) allows us to
recognize the end of an utterance.

As a conclusion, therefore, we propose the following annotation
scheme for dialogue structure based on the structure defined by
Gallardo \cite{Gallardo96}. This proposal is justified on the
studies by Fox \cite{Fox87} about discourse structure and anaphora
resolution:


\begin{description}

\item{{\bf Turn (T)}} is identified by a change of speaker in
the dialogue; each change of speaker supposes a new speaking turn.
On this point, Gallardo makes a distinction between two different
kinds of turns:

\begin{itemize}

\item{An {\bf Intervention Turn (IT)}} is one that adds
information to the dialogue. Such turns constitute what is called
{\it the primary system of conversation}. Speakers use their
interventions to provide information that facilitates the progress
of the topic of conversation. Interventions may be {\bf
initiatives (IT$_{I}$)} when they formulate invitations,
requirements, offers, reports, etc., or {\bf reactions (IT$_{R}$)}
when they answer or evaluate the previous speaker's intervention.
Finally, they can also be {\bf mixed interventions (IT$_{R/I}$)},
meaning a reaction that begins as a response to the previous
speaker's intervention, and ends as an introduction of new
information.

\item{A {\bf Continuing Turn (CT)}} represents an empty turn,
which is quite typical of a listener whose aim is the formal
reinforcement and ratification of the cast of conversational
roles. Such interventions lack information.
\end{itemize}

\item{{\bf Adjacency Pair} or {\bf Exchange (AP)}} is a sequence of turns
 T headed by an initiation intervention turn (IT$_I$) and ended
 by a reaction intervention turn (IT$_R$). One form of anaphora
 which appears to be very common in dialogues is the reference
 within an adjacency pair \cite{Fox87}.

\item{{\bf Topic (TOPIC)} is the main entity in the
dialogue. According to Rocha \cite{Rocha98} four features are
taken into account in the selection of the best candidate for
discourse topic: frequency, even distribution, position of first
token, and semantic adequacy. The topic must be a lexical item
which is frequently referred to.}

\end{description}


According to the above-mentioned structure, the following set of
tags is considered necessary for dialogue structure annotation:
IT$_{I}$, IT$_{R}$, IT$_{R/I}$, CT, AP and TOPIC. AP and TOPIC
tags will be used to define the anaphoric accessibility space and
the remaining will be used to obtain the adjacency pairs.

For this experiment, the corpus has been manually annotated.
However, nowadays there are some works performing an automatic
adjacency pair tagging, such as the {\em Basurde} project
\cite{Basurde98}. On the other hand, there are other works
performing automatic topic tagging (e.g. Reynar \cite{Reynar99})
or automatic topic extraction (e.g. the method for anaphora
resolution shown in Martínez-Barco {\it et al}
\cite{Martinez99a}).

The annotation of conversational dialogues is carried out, as
shown above, and the evaluation of the proposed anaphoric
accessibility space accomplished. An important aspect of this type
of annotation is the training phase, which assures the reliability
of the annotation.

The annotation phase is accomplished in the following way: a) two
annotators are selected, b) an agreement is reached between the
two annotators with regard to the annotation scheme using 5
dialogues (training corpus), c) the annotation is then carried out
by both annotators in parallel over the remaining 35 dialogues
(test corpus) and, d) finally, a reliability test is done on the
annotation (see Carletta {\it et al.} \cite{Carletta97}) in order
to guarantee the results. The reliability test uses the {\it
kappa} statistic that measures the affinity between the
annotations of the two annotators by making judgements about
categories. See Siegel and Castellan \cite{Siegel88} for {\it
kappa} statistic ({\it k}) computing.

According to Carletta, a {\it k} measurement such as $0.68 < k <
0.8$ allows us to make encouraging conclusions, and $k > 0.8$
means total reliability between the results of both annotators.

Once both annotators have carried out the annotation, the
reliability test of the annotation has been run, with a {\it
kappa} measurement of $k=0.91$. We therefore consider the
annotation obtained for the evaluation to be totally reliable.

\section{A brief survey of the algorithm}

Based on the above-mentioned dialogue structure, an algorithm that
combines several kinds of knowledge must be proposed. Concretely,
it must use and combine: a) linguistic knowledge such as lexical,
morphological, syntactic and semantic knowledge; and b) knowledge
from the dialogue structure itself, which is based on the
annotation previously proposed, and knowledge taken from the topic
of the dialogue.

Following the previous directives, an anaphora resolution
algorithm for dialogues was proposed by Palomar and Martínez-Barco
\cite{Palomar2000}. This algorithm was based on the following
three steps:

\begin{enumerate}
\item All possible antecedents are obtained from the structure and topic of dialogue
texts.

\item Linguistic constraints are applied on the previously selected antecedents, with the aim of discarding incompatible antecedents:

\item If there are more than one antecedent left after applying the previous constraints,
 then a set of linguistic and dialogue structure preferences are applied.
\end{enumerate}

\section{Evaluation of the annotation scheme}
In order to carry out the evaluation of this scheme, we had 200
transcribed spoken dialogues ({\em Corpus InfoTren: Person}) at
our disposal, afforded us by the project {\it Basurde}, from which
we selected and POS-tagged 40. The tagging provides lexical and
morphological knowledge for each word of the text. Afterwards, the
structure of the dialogue was manually annotated, following the
directives previously presented. From the 40 POS-tagged dialogues,
5 were randomly selected for the training of the algorithm and the
remaining 35 were reserved in order to carry out the final
evaluation. As a result, 73.8\% precision for pronominal anaphora
resolution and 78.9\% for adjectival anaphora resolution was
achieved.

An empirical study about the precision of the anaphoric
accessibility space using this schema is shown in Martinez-Barco
and Palomar \cite{Martinez2000c}.

\section{Conclusions}
In this paper we have proposed an annotation scheme of dialogue
structure that helps to anaphora resolution in Spanish dialogues.
This annotation scheme has been developed on the basis of works by
Gallardo \cite{Gallardo96} who applied the theory of
conversational analysis to Spanish dialogues and the studies by
Fox \cite{Fox87} about the relationship between this theory and
anaphora. Furthermore, an evaluation of this proposal over an
anaphora resolution algorithm has been shown with successful
results.

\section{Acknowledgments}

The authors wish to thank N. Prieto, F. Pla and A. Molina
(Universitat Politecnica de Valencia) for having contributed their
tagger and E. Segarra (Universitat Politecnica de Valencia) for
affording us the {\em Corpus InfoTren: Person} from the BASURDE
Project.

This paper has been supported by the Spanish Government under
grant HB1998-0068.

\bibliographystyle{plain}
\bibliography{TSDref57}

\end{document}
