NIST Scoring Toolkit Version 2.0beta

Welcome to the NIST Scoring Toolkit Version 2.0 Beta 1

The NIST Scoring Toolkit (SCTK) is a collection of software tools designed to score benchmark test evaluations of Automatic Speech Recognition (ASR) Systems. The toolkit is currently used by NIST, benchmark test participants, and reserchers worldwide to as a common scoring engine.

This is a beta version release of SCTK which now includes the tranfilt software package. The documentation has not been fully integrated because of time limitations.

This version of SCTK contains several programs:

sclite

SCTK has at its core, sclite (Score-Lite), which is a flexible Dynamic Programming alignment engine used to "align" errorful hypothesized texts, such as output from an ASR system, to the correct reference texts. After alignment, sclite generates a veriety of summary as well as detailed scoring reports.

This version of sclite comes bundled with the CMU-Cambridge Statistical Language Modeling Toolkit v2. The toolkit is used to compute word-weights based on an N-gram language model. The directory 'src/slm_v2' contains the complete distribution and is automatically compiled by the installation scripts.

sc_stats

While sclite aligns and scores a single system, sc_stats will compare system performance between more than one system, so long as the systems under test have been ran on identical test data and using an identical test paradigm. Inter-System comparisons are made by running tests paired-comparison statistical significance tests.

rover

Rover - Recognition Output Voting Error Reduction, is a tool which combines an arbitrary number for ASR system outputs into a composite Word Transition network which is then searched an scored to retrieve the best scoring word sequence.

The program is documented in the paper A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER) presented at the 1997 IEEE Workshop on Automatic Speech Recognition and Understanding. The paper is also available in postscript. hubscr

Wrapper script for scoring DARPA STT evaluations

rfilter1

Transcription normalization filter engine

csrfilt

Transcription normalization wrapper script

chfilt

CallHome/Switchboard style transcript filter

hamzaNorm

Arabic Hamza+alif transcript normalization filter

acomp

Automatic compound word expansion filter

def_art

Aribic definate article striping program

utf_filt

UTF file format transcription filter