Hierarchical Regulated Iterative Network for Joint Task of Music Detection and Music Relative Loudness Estimation

Music detection (MD) refers to the task of finding out whether a music event happens in an audio file and what time it starts and ends, i.e., splitting an audio recording and annotating each fragment as music or non-music. MD not only has the basic application in automatic retrieving and localizing audio data based on the type of content but also has a more practical application of monitoring music for copyright management. The practical application in the music industry is the royalty collection in broadcasting. As elaborated in [1]: the Austrian National Broadcasting Corporation (ORF) requires knowing where exactly the music appears in the soundtrack of TV production, and detecting the music is in the foreground or the background. ORF posed this requirement for the purpose of calculating the royalty fees, which are paid to a national agency according to certain rules. Ideally, the production team would provide a list of all the music segments occurring in TV production, but in reality, these lists are largely inaccurate. As a result, ORF has to guess the amount of music within a production more or less because manually annotating all productions is impossible. Also, the copyright fee will be different for music is used in the foreground or the background [2]. Hence, it is highly expected to develop method of music relative loudness estimation (MRLE), i.e., annotating each fragment as fg-music, bg-music, or non-music.

In the past, research mainly focused on the music/speech detection task, which is segmenting and annotating audio as music, speech, or noise. Early work [3] explored the distinguishable features between music and speech from the perspective of signal processing. Using these handcrafted features, later research [4]–[5][6] added subsequent classifiers to do music/speech detection. Recent works [7]–[8][9] focused on automatically-learned features from spectrogram images and used neural networks as classifiers. In contrast to simple music/speech detection task, the emphasis point of MD task is different: music is used to accentuate scenes, therefore speech and any noise signals might be present concurrently.