End-to-end Evaluation in JANUS: A Speech-to-speech Translation System

August 1996

End-to-end Evaluation in JANUS: A Speech-to-speech Translation System

Authors:

Donna Gates, Alon Lavie, Lori Levin, Alex Waibel, Marsal Gavalda, Monika Woszczyna, and Puming Zhan

Abstract:

JANUS is a multi-lingual speech-to-speech translation system designed to facilitate communication between two parties engaged in a spontaneous conversation in a limited domain. In this paper we describe our methodology for evaluating translation performance. Our current focus is on end- to- end evaluations- the evaluation of the translation capabilities of the system as a whole. The main goal of our end-to-end evaluation procedure is to determine translation accuracy on a test set of previously unseen dialogues. Other goals include evaluating the effectiveness of the system in conveying domain-relevant information and in detecting and dealing appropriately with utterances (or portions of utterances) that are out-of-domain. End-to-end evaluations are performed in order to verify the general coverage of our knowledge sources, guide our development efforts, and to track our improvement over time. We discuss our evaluation procedures, the criteria used for assigning scores to translations produced by the system, and the tools developed for performing this task. Recent Spanish-to-English performance evaluation results are presented as an example.
@workshop{Gates-1996-16276,
author = {Donna Gates And Alon Lavie And Lori Levin And Alex Waibel And Marsal Gavalda And And Monika Woszczyna And Puming Zhan},
title = {End-to-end Evaluation in JANUS: A Speech-to-speech Translation System},
booktitle = {Proceedings of ECAI '96 Workshop on Dialogue Processing in Spoken Language Systems (DPSLS '96)},
year = {1996},
month = {August},
pages = {195 - 206},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.