Abstract Human spoken communication involves not only lexical content but also non-verbal vocalizations (NVs) such as laughter, sighs, and coughs, which convey emotions, intentions, and social signals. However, most existing speech systems focus solely on verbal content and lack the ability to understand and generate such non-verbal cues, reducing the emotional intelligence and communicative richness of spoken interfaces. In this work, we introduce NonVerbalSpeech-38K, a large and diverse dataset for non-verbal speech generation and understanding, collected from real-world media and annotated using an automatic pipeline. The dataset contains 38,718 samples (about 131 hours) with 10 categories of non-verbal cues, such as laughter, sniff, and throat clearing. We further validate the dataset by fine-tuning state-of-the-art models, including F5-TTS and Qwen2-Audio, demonstrating its effectiveness in non-verbal speech generation and understanding tasks. Our contributions are threefold: (1) We propose a practical pipeline for building natural and diverse non-verbal speech datasets; (2) We release a large-scale dataset to advance research on non-verbal speech generation and understanding; (3) We validate the dataset’s effectiveness by demonstrating improvements in both non-verbal speech synthesis and captioning, thereby facilitating richer human-computer interaction..

Contents

• Pipeline Overview
• Non-Verbal Speech Generation (ZH)
• Non-Verbal Speech Generation (EN)
• Non-Verbal Speech Understanding
• Dataset Samples

This page is for research demonstration purposes only.

Pipeline Overview

Figure 1. An overview of our NonVerbalSpeech-38K pipeline. It consists of three main stages: (1) Data Preparation and Preprocessing, (2) Non-Verbal Segment Detection, and (3) Integration of Non-Verbal Tags into Speech Content. Among these, NonVerbal Segment Detection serves as the core component of the pipeline.

Figure 2. Data preparation and frame-level detection model training.

Non-Verbal Speech Generation (ZH)

Note: Dia appears to lack support for Chinese (ZH), resulting in unintelligible audio. Furthermore, the pretrained F5-TTS model does not support non-verbal expression control.

Non-Verbal Speech Generation (EN)

Non-Verbal Speech Understanding

NonVerbalSpeech-38K dataset samples

More information about our NonVerbalSpeech-38K dataset is available on Hugging Face.

Note:

• The <B>; and </B> tags indicate that non-verbal expressions overlap with spoken words.
• Due to limitations in ASR-based timestamps, slight misalignments may exist between the audio and captions. However, the detected non-verbal segments themselves are accurate. We plan to enhance alignment quality in future updates.
• In the current version, we extend overlapping non-verbal expressions to the sub-sentence level to reduce misalignment issues. For example, we modify:
  李渊看着秦琼心说，这是我的金殿哪，他在这儿就指着我两个儿子，让我两个儿子心服口服。哎呀。李渊心说，大唐[sniff]<B>江山</B>要紧，我也不能再包庇我的两个儿子了。哎呀，是世民，你也给我跪下。
  to:
  李渊看着秦琼心说，这是我的金殿哪，他在这儿就指着我两个儿子，让我两个儿子心服口服。哎呀。李渊心说，[sniff]<B>大唐江山要紧</B>，我也不能再包庇我的两个儿子了。哎呀，是世民，你也给我跪下。
  This strategy aims to improve alignment between captions and audio. However, we observe that such modifications do not significantly enhance alignment quality.
  To support further research, we also release the Non-Verbal Regions. detected by our model, which can be used to re-annotate the dataset with more accurate timestamps.