Representing a corpus

In Lhotse, we represent the data using a small number of Python classes, enhanced with methods for solving common data manipulation tasks, that can be stored as JSON or JSONL manifests. For most audio corpora, we will need two types of manifests to fully describe them: a recording manifest and a supervision manifest.

Recording manifest

class lhotse.audio.Recording(id, sources, sampling_rate, num_samples, duration, channel_ids=None, transforms=None)

The Recording manifest describes the recordings in a given corpus. It contains information about the recording, such as its path(s), duration, the number of samples, etc. It allows to represent multiple channels coming from one or more files.

This manifest does not specify any segmentation information or supervision such as the transcript or the speaker – we use SupervisionSegment for that.

Note that Recording can represent both a single utterance (e.g., in LibriSpeech) and a 1-hour session with multiple channels and speakers (e.g., in AMI). In the latter case, it is partitioned into data suitable for model training using Cut.

Internally, Lhotse supports multiple audio backends to read audio file. By default, we try to use libsoundfile, then torchaudio (with FFMPEG integration starting with torchaudio 2.1), and then audioread (which is an ffmpeg CLI wrapper). For sphere files we prefer to use sph2pipe binary as it can work with certain unique encodings such as “shorten”.

Audio backends in Lhotse are configurable. See:

• available_audio_backends()

• audio_backend(),

• get_current_audio_backend()

• set_current_audio_backend()

• get_default_audio_backend()

Examples

A Recording can be simply created from a local audio file:

>>> from lhotse import RecordingSet, Recording, AudioSource
>>> recording = Recording.from_file('meeting.wav')
>>> recording
Recording(
    id='meeting',
    sources=[AudioSource(type='file', channels=[0], source='meeting.wav')],
    sampling_rate=16000,
    num_samples=57600000,
    duration=3600.0,
    transforms=None
)

This manifest can be easily converted to a Python dict and serialized to JSON/JSONL/YAML/etc:

>>> recording.to_dict()
{'id': 'meeting',
 'sources': [{'type': 'file',
   'channels': [0],
   'source': 'meeting.wav'}],
 'sampling_rate': 16000,
 'num_samples': 57600000,
 'duration': 3600.0}

Recordings can be also created programatically, e.g. when they refer to URLs stored in S3 or somewhere else:

>>> s3_audio_files = ['s3://my-bucket/123-5678.flac', ...]
>>> recs = RecordingSet.from_recordings(
...     Recording(
...         id=url.split('/')[-1].replace('.flac', ''),
...         sources=[AudioSource(type='url', source=url, channels=[0])],
...         sampling_rate=16000,
...         num_samples=get_num_samples(url),
...         duration=get_duration(url)
...     )
...     for url in s3_audio_files
... )

It allows reading a subset of the audio samples as a numpy array:

>>> samples = recording.load_audio()
>>> assert samples.shape == (1, 16000)
>>> samples2 = recording.load_audio(offset=0.5)
>>> assert samples2.shape == (1, 8000)

See also: Recording, Cut, CutSet.

class lhotse.audio.RecordingSet(recordings=None)

RecordingSet represents a collection of recordings. It does not contain any annotation such as the transcript or the speaker identity – just the information needed to retrieve a recording such as its path, URL, number of channels, and some recording metadata (duration, number of samples).

It also supports (de)serialization to/from YAML/JSON/etc. and takes care of mapping between rich Python classes and YAML/JSON/etc. primitives during conversion.

When coming from Kaldi, think of it as wav.scp on steroids: RecordingSet also has the information from reco2dur and reco2num_samples, is able to represent multi-channel recordings and read a specified subset of channels, and support reading audio files directly, via a unix pipe, or downloading them on-the-fly from a URL (HTTPS/S3/Azure/GCP/etc.).

Examples:

RecordingSet can be created from an iterable of Recording objects:

>>> from lhotse import RecordingSet
>>> audio_paths = ['123-5678.wav', ...]
>>> recs = RecordingSet.from_recordings(Recording.from_file(p) for p in audio_paths)

As well as from a directory, which will be scanned recursively for files with parallel processing:

>>> recs2 = RecordingSet.from_dir('/data/audio', pattern='*.flac', num_jobs=4)

It behaves similarly to a dict:

>>> '123-5678' in recs
True
>>> recording = recs['123-5678']
>>> for recording in recs:
>>>    pass
>>> len(recs)
127

It also provides some utilities for I/O:

>>> recs.to_file('recordings.jsonl')
>>> recs.to_file('recordings.json.gz')  # auto-compression
>>> recs2 = RecordingSet.from_file('recordings.jsonl')

Manipulation:

>>> longer_than_5s = recs.filter(lambda r: r.duration > 5)
>>> first_100 = recs.subset(first=100)
>>> split_into_4 = recs.split(num_splits=4)
>>> shuffled = recs.shuffle()

And lazy data augmentation/transformation, that requires to adjust some information in the manifest (e.g., num_samples or duration). Note that in the following examples, the audio is untouched – the operations are stored in the manifest, and executed upon reading the audio:

>>> recs_sp = recs.perturb_speed(factor=1.1)
>>> recs_vp = recs.perturb_volume(factor=2.)
>>> recs_rvb = recs.reverb_rir(rir_recs)
>>> recs_24k = recs.resample(24000)

Supervision manifest

class lhotse.supervision.SupervisionSegment(id, recording_id, start, duration, channel=0, text=None, language=None, speaker=None, gender=None, custom=None, alignment=None)

SupervisionSegment represents a time interval (segment) annotated with some supervision labels and/or metadata, such as the transcription, the speaker identity, the language, etc.

Each supervision has unique id and always refers to a specific recording (via recording_id) and one or more channel (by default, 0). Note that multiple channels of the recording may share the same supervision, in which case the channel field will be a list of integers.

It’s also characterized by the start time (relative to the beginning of a Recording or a Cut) and a duration, both expressed in seconds.

The remaining fields are all optional, and their availability depends on specific corpora. Since it is difficult to predict all possible types of metadata, the custom field (a dict) can be used to insert types of supervisions that are not supported out of the box.

SupervisionSegment may contain multiple types of alignments. The alignment field is a dict, indexed by alignment’s type (e.g., word or phone), and contains a list of AlignmentItem objects – simple structures that contain a given symbol and its time interval. Alignments can be read from CTM files or created programatically.

Examples

A simple segment with no supervision information:

>>> from lhotse import SupervisionSegment
>>> sup0 = SupervisionSegment(
...     id='rec00001-sup00000', recording_id='rec00001',
...     start=0.5, duration=5.0, channel=0
... )

Typical supervision containing transcript, speaker ID, gender, and language:

>>> sup1 = SupervisionSegment(
...     id='rec00001-sup00001', recording_id='rec00001',
...     start=5.5, duration=3.0, channel=0,
...     text='transcript of the second segment',
...     speaker='Norman Dyhrentfurth', language='English', gender='M'
... )

Two supervisions denoting overlapping speech on two separate channels in a microphone array/multiple headsets (pay attention to start, duration, and channel):

>>> sup2 = SupervisionSegment(
...     id='rec00001-sup00002', recording_id='rec00001',
...     start=15.0, duration=5.0, channel=0,
...     text="i have incredibly good news for you",
...     speaker='Norman Dyhrentfurth', language='English', gender='M'
... )
>>> sup3 = SupervisionSegment(
...     id='rec00001-sup00003', recording_id='rec00001',
...     start=18.0, duration=3.0, channel=1,
...     text="say what",
...     speaker='Hervey Arman', language='English', gender='M'
... )

A supervision with a phone alignment:

>>> from lhotse.supervision import AlignmentItem
>>> sup4 = SupervisionSegment(
...     id='rec00001-sup00004', recording_id='rec00001',
...     start=33.0, duration=1.0, channel=0,
...     text="ice",
...     speaker='Maryla Zechariah', language='English', gender='F',
...     alignment={
...         'phone': [
...             AlignmentItem(symbol='AY0', start=33.0, duration=0.6),
...             AlignmentItem(symbol='S', start=33.6, duration=0.4)
...         ]
...     }
... )

A supervision shared across multiple channels of a recording (e.g. a microphone array):

>>> sup5 = SupervisionSegment(
...     id='rec00001-sup00005', recording_id='rec00001',
...     start=33.0, duration=1.0, channel=[0, 1],
...     text="ice",
...     speaker='Maryla Zechariah',
... )

Converting SupervisionSegment to a dict:

>>> sup0.to_dict()
{'id': 'rec00001-sup00000', 'recording_id': 'rec00001', 'start': 0.5, 'duration': 5.0, 'channel': 0}

class lhotse.supervision.SupervisionSet(segments=None)

SupervisionSet represents a collection of segments containing some supervision information (see SupervisionSegment).

It acts as a Python list, extended with an efficient find operation that indexes and caches the supervision segments in an interval tree. It allows to quickly find supervision segments that correspond to a specific time interval. However, it can also work with lazy iterables.

When coming from Kaldi, think of SupervisionSet as a segments file on steroids, that may also contain text, utt2spk, utt2gender, utt2dur, etc.

Examples

Building a SupervisionSet:

>>> from lhotse import SupervisionSet, SupervisionSegment
>>> sups = SupervisionSet.from_segments([SupervisionSegment(...), ...])

Writing/reading a SupervisionSet:

>>> sups.to_file('supervisions.jsonl.gz')
>>> sups2 = SupervisionSet.from_file('supervisions.jsonl.gz')

Using SupervisionSet like a dict:

>>> 'rec00001-sup00000' in sups
True
>>> sups['rec00001-sup00000']
SupervisionSegment(id='rec00001-sup00000', recording_id='rec00001', start=0.5, ...)
>>> for segment in sups:
...     pass

Searching by recording_id and time interval:

>>> matched_segments = sups.find(recording_id='rec00001', start_after=17.0, end_before=25.0)

Manipulation:

>>> longer_than_5s = sups.filter(lambda s: s.duration > 5)
>>> first_100 = sups.subset(first=100)
>>> split_into_4 = sups.split(num_splits=4)
>>> shuffled = sups.shuffle()

Standard data preparation recipes

We provide a number of standard data preparation recipes. By that, we mean a collection of a Python function + a CLI tool that create the manifests given a corpus directory.

Currently supported audio corpora

Corpus name	Function
ADEPT	lhotse.recipes.prepare_adept()
Aidatatang_200zh	lhotse.recipes.prepare_aidatatang_200zh()
Aishell	lhotse.recipes.prepare_aishell()
Aishell-3	lhotse.recipes.prepare_aishell3()
AISHELL-4	lhotse.recipes.prepare_aishell4()
AliMeeting	lhotse.recipes.prepare_alimeeting()
AMI	lhotse.recipes.prepare_ami()
ASpIRE	lhotse.recipes.prepare_aspire()
ATCOSIM	lhotse.recipes.prepare_atcosim()
AudioMNIST	lhotse.recipes.prepare_audio_mnist()
BABEL	lhotse.recipes.prepare_single_babel_language()
Bengali.AI Speech	lhotse.recipes.prepare_bengaliai_speech()
BUT ReverbDB	lhotse.recipes.prepare_but_reverb_db()
BVCC / VoiceMOS Challenge	lhotse.recipes.bvcc()
CallHome Egyptian	lhotse.recipes.prepare_callhome_egyptian()
CallHome English	lhotse.recipes.prepare_callhome_english()
CHiME-6	lhotse.recipes.prepare_chime6()
CMU Arctic	lhotse.recipes.prepare_cmu_arctic()
CMU Indic	lhotse.recipes.prepare_cmu_indic()
CMU Kids	lhotse.recipes.prepare_cmu_kids()
CommonVoice	lhotse.recipes.prepare_commonvoice()
Corpus of Spontaneous Japanese	lhotse.recipes.prepare_csj()
CSLU Kids	lhotse.recipes.prepare_cslu_kids()
DailyTalk	lhotse.recipes.prepare_daily_talk()
DIHARD III	lhotse.recipes.prepare_dihard3()
DiPCo	lhotse.recipes.prepare_dipco()
Earnings’21	lhotse.recipes.prepare_earnings21()
Earnings’22	lhotse.recipes.prepare_earnings22()
The Edinburgh International Accents of English Corpus	lhotse.recipes.prepare_edacc()
English Broadcast News 1997	lhotse.recipes.prepare_broadcast_news()
Fisher English Part 1, 2	lhotse.recipes.prepare_fisher_english()
Fisher Spanish	lhotse.recipes.prepare_fisher_spanish()
Fluent Speech Commands	lhotse.recipes.slu()
GALE Arabic Broadcast Speech	lhotse.recipes.prepare_gale_arabic()
GALE Mandarin Broadcast Speech	lhotse.recipes.prepare_gale_mandarin()
GigaSpeech	lhotse.recipes.prepare_gigaspeech()
GigaST	lhotse.recipes.prepare_gigast()
Heroico	lhotse.recipes.prepare_heroico()
HiFiTTS	lhotse.recipes.prepare_hifitts()
HI-MIA (including HI-MIA-CW)	lhotse.recipes.prepare_himia()
ICMC-ASR	lhotse.recipes.prepare_icmcasr()
ICSI	lhotse.recipes.prepare_icsi()
IWSLT22_Ta	lhotse.recipes.prepare_iwslt22_ta()
KeSpeech	lhotse.recipes.prepare_kespeech()
L2 Arctic	lhotse.recipes.prepare_l2_arctic()
LibriCSS	lhotse.recipes.prepare_libricss()
LibriLight	lhotse.recipes.prepare_librilight()
LibriSpeech (including “mini”)	lhotse.recipes.prepare_librispeech()
LibriTTS	lhotse.recipes.prepare_libritts()
LibriTTS-R	lhotse.recipes.prepare_librittsr()
LJ Speech	lhotse.recipes.prepare_ljspeech()
MDCC	lhotse.recipes.prepare_mdcc()
Medical	lhotse.recipes.prepare_medical()
MiniLibriMix	lhotse.recipes.prepare_librimix()
MTEDx	lhotse.recipes.prepare_mtdex()
MobvoiHotWord	lhotse.recipes.prepare_mobvoihotwords()
Multilingual LibriSpeech (MLS)	lhotse.recipes.prepare_mls()
MUSAN	lhotse.recipes.prepare_musan()
MuST-C	lhotse.recipes.prepare_must_c()
National Speech Corpus (Singaporean English)	lhotse.recipes.prepare_nsc()
People’s Speech	lhotse.recipes.prepare_peoples_speech()
RIRs and Noises Corpus (OpenSLR 28)	lhotse.recipes.prepare_rir_noise()
Speech Commands	lhotse.recipes.prepare_speechcommands()
SpeechIO	lhotse.recipes.prepare_speechio()
SPGISpeech	lhotse.recipes.prepare_spgispeech()
Switchboard	lhotse.recipes.prepare_switchboard()
TED-LIUM v2	lhotse.recipes.prepare_tedlium2()
TED-LIUM v3	lhotse.recipes.prepare_tedlium()
TIMIT	lhotse.recipes.prepare_timit()
This American Life	lhotse.recipes.prepare_this_american_life()
UWB-ATCC	lhotse.recipes.prepare_uwb_atcc()
VCTK	lhotse.recipes.prepare_vctk()
VoxCeleb	lhotse.recipes.prepare_voxceleb()
VoxConverse	lhotse.recipes.prepare_voxconverse()
VoxPopuli	lhotse.recipes.prepare_voxpopuli()
WenetSpeech	lhotse.recipes.prepare_wenet_speech()
YesNo	lhotse.recipes.prepare_yesno()
Eval2000	lhotse.recipes.prepare_eval2000()
MGB2	lhotse.recipes.prepare_mgb2()
XBMU-AMDO31	lhotse.recipes.xbmu_amdo31()

Currently supported video corpora

Corpus name	Function
Grid Audio-Visual Speech Corpus	lhotse.recipes.prepare_grid()

Adding new corpora

Hint

Python data preparation recipes. Each corpus has a dedicated Python file in lhotse/recipes, which you can use as the basis for your own recipe.

Hint

(optional) Downloading utility. For publicly available corpora that can be freely downloaded, we usually define a function called download_<corpus-name>().

Hint

Data preparation Python entry-point. Each data preparation recipe should expose a single function called prepare_<corpus-name>, that produces dicts like: {'recordings': <RecordingSet>, 'supervisions': <SupervisionSet>}.

Hint

CLI recipe wrappers. We provide a command-line interface that wraps the download and prepare functions – see lhotse/bin/modes/recipes for examples of how to do it.

Hint

Pre-defined train/dev/test splits. When a corpus defines standard split (e.g. train/dev/test), we return a dict with the following structure: {'train': {'recordings': <RecordingSet>, 'supervisions': <SupervisionSet>}, 'dev': ...}

Hint

Manifest naming convention. The default naming convention is <corpus-name>_<manifest-type>_<split>.jsonl.gz, i.e., we save the manifests in a compressed JSONL file. Here, <manifest-type> can be recordings, supervisions, etc., and <split> can be train, dev, test, etc. In case the corpus has no such split defined, we can use all as default. Other information, e.g., mic type, language, etc. may be included in the <corpus-name>. Some examples are: cmu-indic_recordings_all.jsonl.gz, ami-ihm_supervisions_dev.jsonl.gz, mtedx-english_recordings_train.jsonl.gz.

Hint

Isolated utterance corpora. Some corpora (like LibriSpeech) come with pre-segmented recordings. In these cases, the SupervisionSegment will exactly match the Recording duration (and there will likely be exactly one segment corresponding to any recording).

Hint

Conversational corpora. Corpora with longer recordings (e.g. conversational, like Switchboard) should have exactly one Recording object corresponding to a single conversation/session, that spans its whole duration. Each speech segment in that recording should be represented as a SupervisionSegment with the same recording_id value.

Hint

Multi-channel corpora. Corpora with multiple channels for each session (e.g. AMI) should have a single Recording with multiple AudioSource objects – each corresponding to a separate channel. Remember to make the SupervisionSegment objects correspond to the right channels!