A Multimodal Approach for Audivisual Data Segmentation and Annotation

Abstract

While most approaches for video segmentation and indexing are focused on the pictorial part, there are significant clues contained in the accompanying audio flow. Only by combining the audio and visual information together, a fully functional system for video content parsing can be achieved. Based on the investigation of data structures for different video types, we present in this paper a scheme for the segmentation and annotation of audiovisual sequence which includes tools for both audio and visual content analysis. In the proposed system, the video data is segmented into audio scenes and visual shots by detecting abrupt changes in the audio and visual features, respectively. Then, the audio scene is indexed as one of the basic audio types such as speech, music, song, environmental sound, speech with music background, etc. while the visual shot is represented by keyframes and associated image features. An index table is generated automatically for each video clip through the integration of audio and visual analysis outputs. Experimental results show that the proposed research accomplishes more meaningful and robust indexing of video content compared to previous work.