MPEG-7 has emerged as the standard for multimedia data content description. As it is in his early age, it tries to evolve in a direction in which semantic content description can be implemented. Although many descriptors (Ds) and description schemes (DSs) provided by the MPEG-7 standard can help to implement semantics of a media, grouping together several mpeg-7 classes can provide better results in the video production and video analysis tasks.

Our Method

We provide some classes to extend the mpeg-7 standard so it can handle, in a more uniform way, the video media data. Several classes are proposed in this context and we prove that this kind of schemes can provide more flexible tools

By those new descriptors we achieve:

• An Anthropocentric Perspective for Movies.
• We introduce Descriptors and Description Schemes, in order to manipulate in better way low level information, and thus provide semantic entities.
• The relations between objects within a movie are very informative for high level information extraction.
• Information like “This actor is in this shot and smiling” can be ingested in the proposed profile and in a post process of this information one can extract semantics for this shot.

Main Characteristics

• Descriptors (Ds) and Description Schemes (DSs) which are gathering low information within their tags.
• Tags which are selected to suit research areas like face detection, object tracking, motion detection, facial expression extraction etc.
• The organization of the aforementioned low level information will produce objects with meaning in order to extract high level information

The Descriptors and Description Schemes

Anthropocentric View

• Simple annotation can provide information about everything but it can not be generated automatically.
• The annotation process is therefore subjective as in all manual annotation processes and also demands for an intensive labor.
• The proposed profile aims at providing support for combining calculated low level features into higher level semantic entities.

Downloads

-

Relevant Publications

N. Vretos, V. Solachidis and I. Pitas, "An Anthropocentric Description Scheme For Movies Content Classification And Indexing" , in Proc. of European Signal Processing Conf. (EUSIPCO 2005) , Antalya, Turkey, 4-8 September, 2005.

N. Vretos, V. Solachidis and I. Pitas, "An MPEG-7 Based Description Scheme For Video Analysis using Anthropocentric Video Content Descriptors", in Lecture Notes in Computer Science, Advances in Informatics: 10th Panhellenic Conf. on Informatics, PCI 2005 , vol. 3746 / 2005, pp. 725 - 734, Volos, Greece, 11-13 November, 2005.

Research Projects

NM2 - “New media for a new millennium” (IST-004124), FP6S

Achieving a good localization of faces on video frames is of high importance for an application such as video indexing. Face localization on movies is an ambiguous task due to various scale, pose and lighting conditions.

Our Method

A novel deterministic approach has been developed that it applies face detection, forward tracking and backward tracking, using some predefined rules. From all the possible extracted candidates, a Dynamic Programming algorithm selects those that minimize a cost function.

Face detection:

Use of the Haar-like features to provide first face candidates. For tracking purposes, a post-processing step is added to reduce the number of false alarms and remove some of the background. Candidates are rejected if the number of pixels fulfilling the criteria below are under a certain threshold.

0 < h < 1 and 0.23 < s < 0.68 and 0.27 < v

The remaining candidates are replaced by the smallest bounding box containing the skin-like pixels. The detection is performed every 5 frames.

Forward tracking process:

Backward tracking process:

Clustering could be considered as a form of unsupervised classification imposed over a finite set of objects. Its goal is to group sets of objects into classes, such that similar objects are placed in the same cluster, while dissimilar objects are placed in different clusters.

Human faces are some of the most important and frequently encountered entities in videos and can be considered as high-level semantic features. Face clustering in videos can be used in many applications such as video indexing and content analysis, as a pre processing step for face recognition, or even as a basic step for extracting the principal cast of a feature length movie and much more.

.

Our Methods

A) Clustering based on Mutual Information: The capabilities of joint entropy and mutual information are exploited in order to classify face images exported from a Haar detector. We use the intensity images and we define for every image the probability density function as the histogram of the intensities of that image summed to one. In order to calculate the joint entropy between the two images we construct a 2D histogram of 256 bins which take into account the relative positions of intensities so that similarity occurs between two images, when same intensities are located in same spatial locations.

Problem Definition

• Face clustering is the task where from a set of face images A we create n subsets
  
• We exploit the mutual information between the face images to create a similarity matrix which afterwards will be clustered.
• The mutual information is shown to be a good measure for similarity between face images where light conditions and poses are variant.
• The movies' context for such an algorithm gives a new dimension to the problem where no calibrated images are used as input.
• Purpose of such an algorithm: Define primordial actors, automatic (not manually annotated) database search, registration, content analysis.

Clustering Process:

• The clustering process is based on the Fuzzy-C Means (FCM) algorithm.
• We provide the number of classes and the similarity matrix to the algorithm.
• In order to use this algorithm we define every row of the aforementioned similarity matrix as a different vector in an M-dimensional L2-normed vector space over R.

Darker regions belong to the first actor and clearer ones to the second actor. The video sequence has four consecutive shots in the order FA-FA-SA-SA where FA and SA first and second actor respectively

.

B) Hierarchical Clustering: An algorithm to cluster face images found in feature length movies and generally in video sequences is proposed. A novel method for creating a dissimilarity matrix using SIFT image features is introduced. This dissimilarity matrix is used as an input in a hierarchical average linkage clustering algorithm, which finally yields the clustering result.

The final result is found to be quite robust to significant scale, pose and illumination variations, encountered in facial images.

Clusters 1, 2 and 4 contained only facial images from the same person. The third cluster contained the false face detections (non-facial images) as we expected, but it also included certain instances of the actor in cluster 1, due to a significant change in the person's pose.

.

Downloads

-

Relevant Publications

Ν. Vretos, V. Solachidis and I. Pitas "A Mutual Information Based Algorithm for Face Clustering", in Proc. of Int. Conf. on Multimedia and Expo (ICME 2006) , Toronto Ontario, Canada, 9-12 July, 2006.

P. Antonopoulos, N. Nikolaidis and I. Pitas, “Hierarchical Face Clustering Using SIFT Image Features”, submitted in Proc. of IEEE Symposium on Computational Intelligence in Image and Signal Processing (CIISP 2007), Honolulu, HI , USA.

Research Projects

NM2 - “New media for a new millennium” (IST-004124), FP6

Pythagoras II - Funded by the Hellenic Ministry of Education in the framework of the program

Tracking rigid objects and object features in video sequences, is a frequently encountered task in many video-based applications that include surveillance, video editing, virtual reality and computer graphics, human-computer interaction and 3D scene reconstruction from uncalibrated video. It is obvious that building a tracking system is far from being a simple process due to varying lighting conditions, partial occlusions, clutter, unconstrained motion, etc.

Our Method

A novel approach for selecting and tracking feature points in 2D images has been developed. The method selects salient feature points within a given region and proceeds in tracking them over the frames of a video sequence. In this method:

• The image intensity is represented by a 3D deformable surface model
• Selecting and tracking are performed by exploiting a by-product of explicit surface deformation governing equations

The proposed method was compared with the well known Kanade-Lucas-Tomasi (KLT) tracking algorithm, in terms of tracking accuracy and robustness. The obtained results show the superiority of the proposed method.

The initial facial image, its surface representation and the deformed model which wraps image intensity surface

300 feature points were selected on a facial image

Downloads

-

Relevant Publications

M. Krinidis, N. Nikolaidis and I. Pitas, "Feature-based tracking using 3D physics-based deformable surfaces", in Proc. of 2005 EURASIP European Signal Processing Conf. (EUSIPCO 2005), Antalya, Turkey, 4-8 September, 2005.

L. Goldmann, M. Krinidis, N. Nikolaidis, S. Asteriadis and T. Sikora, "An Integrated System for Face Detection and Tracking", in Proc. of Workshop On Immersive Communication And Broadcast Systems (ICOB 2005), Berlin, Germany, 27-28 October, 2005.

Research Projects

VISNET -  Networked Audiovisual Media Technologies, IST, FP6

Indexing and retrieval of digital video is a very active research area. Temporal video segmentation is an important step in many video processing applications. The growing amount of digital video footage is driving the need for more effective methods for shot classification, summarization, efficient access, retrieval, and browsing of large video databases. Shot boundary detection is the first step towards further analysis of the video content.

Our Method

Two methods for shot boundary detection have been developed.

The first approach to shot transition detection in the uncompressed image domain, we have developed, is based on the mutual information and the joint entropy between two consecutive video frames.

• Mutual information (MI) is a measure of the information transported from one frame to the next.
• MI is used within the context of this method for detecting abrupt cuts, where the image intensity or color changes abruptly, leading to a low mutual information value.
• Joint entropy is used for detecting fades.
  • Fade-out, where the visual intensity is usually decreasing to a black image, the decreasing inter-frame joint entropy is used for detection.
  • Fade-in, the increasing joint entropy is used for detection.
• The entropy measure produces good results, because it exploits the interframe information flow in a more compact way than a frame subtraction.

Time series of the MI from “ABC news” video sequence showing abrupt cuts and one fade		The joint entropy signal from “CNN news” video sequence showing a fade-out and fade-in to the next shot

The detection technique was tested on the TRECVID2003 video test set having different types of shots and containing significant object and camera motion inside the shots. The application of these entropy-based techniques for shot cut detection was experimentally proven to be very efficient, since they produce false acceptance rates very close to zero.

The second approach to automated shot boundary detection is using singular value decomposition (SVD). We have used SVD for its capabilities to derive a refined low dimensional feature space from a high dimensional raw feature space, where pattern similarity can easily be detected.

• The method relies on performing SVD on a matrix created from 3D color histograms of single frames.
• After performing SVD we preserved only the 10 largest singular values.
• In order to detect the video shots, the feature vectors from SVD are processed using a dynamic clustering method.
• To avoid the false detections, every two consecutive clusters, obtained by the clustering procedure are in the second phase tested for a possible merging.
• Merging is performed in two steps applied consecutively.
  • The fist step is using ratio cosine similarity measure between clusters.
  • The second step is based on statistical hypothesis testing using the von Mises-Fisher distribution, which can be considered as the equivalent of the Gaussian distribution for directional data.
    
    

Projected frame histograms on the subspace defined by the fifth and sixth singular vectors reveal a dissolve pattern between two shots		Fade detection in the sequence “basketball” visualized on the subspace defined by the first and second left singular vectors

The method can detect cuts and gradual transitions, such as dissolves, fades and wipes. The detection technique was tested on TV video sequences having various types of shots and significant object and camera motion inside the shots. The experiments demonstrated that, by using the projected feature space we can efficiently differentiate between gradual transitions and cuts, pans, object or camera motion, while most of the methods based on histograms fail to characterize these types of video transitions.

Downloads

-

Relevant Publications

Z. Cernekova, I. Pitas and C. Nikou, "Information theory-based shot cut/fade detection and video summarization", IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, no.1, page(s): 82- 91, January 2006.

Z.Cernekova, C.Kotropoulos and I.Pitas, "Video Shot Segmentation using Singular Value Decomposition", in Proc. of 2003 IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), vol. III, pp. 181-184, Hong-Kong, April 2003 (appears also in Proc. IEEE Multimedia and Expo 2003 (ICME), pp. 301-304, Baltimore , July 2003).

Z.Cernekova, C.Kotropoulos and I.Pitas, "Video Shot Boundary Detection using Singular Value Decomposition", in Proc. of 4th European Workshop on Image Analysis for Multimedia Interactive Services(WIAMIS-2003), London, April 2003.

Research Projects

MOUMIR - "Models for Unified Multimedia Information Retrieval", RTN, EC

MUSCLE - “Multimedia Understanding through Semantics, Computation and LEarning” (FP6-507752)

VISNET - European Network of Excellence, funded under the European Commission IST FP6 programme

COST211 - "Redundancy Reduction Techniques and Content Analysis for Multimedia Services"

Histogram equalization is a simple and effective method for image contrast manipulation. It aims together with other methods, such as noise reduction, edge crispening and sharpening, filtering, pseudocoloring, image interpolation and magnification to enhance images either from the human visual perspective or for their effective use in several applications.

Histogram equalization becomes a tedious task when dealing with color images due to:

• the vectorial nature of color: each color pixel is represented by a vector with as many components as the color components in a proper color space (i.e., the three components Red, Green, and Blue in the RGB space)
• the correlation between the color components
• the color perception by humans.

Our Method

A novel color image histogram equalization method has been developed that

• works on the HSI color space leaving Hue unmodified
• exploits the notion of unigram and bigram probabilities borrowed from statistical language modeling
• conducts probability smoothing by an absolute discounting back-off technique (before the equalization process)

in order to respectively

• preserve color information
• jointly equalize the Saturation and intensity Components
• deal with the unseen color component combinations stemming from the dimensionality of the color space and the limited number of colors present in an image .

The method has been furtherly extended by an empirical gamut elimination technique based on the transformations proposed by Naik and Murhy in order to deal with colors in the equalized image that lie outside the color space gamut.

Downloads

-

Relevant Publications

N.Bassiou and C. Kotropoulos, “Color Image Histogram Equalization by Absolute Discounting Back-off, accepted for publication in Elsevier Computer Vision and Image Understanding, February 2007.

N.Bassiou and C. Kotropoulos, “Color Histogram Equalization using Probability Smoothing”, in Proc. of European Signal Processing Conference (EUSIPCO 2006), Florence, Italy, 4-8 September, 2006.

Research Projects

MUSCLE - “Multimedia Understanding through Semantics, Computation and LEarning” (FP6-507752)

A content-based video indexing module has been developed that aims at temporally indexing a video sequence according to the actual speaker. This is achieved by the integration of audio and visual information. Audio analysis leads to the extraction of a speaker identity label versus time diagram. Visual analysis includes scene cut detection, face shot determination, mouth region extraction and tracking and finally talking face shot determination. Results from both sources are combined to improve speaker-dependent video indexing.

Our Module

Audio Processing Module

• Audio Feature Extraction: LP Cepstal Coefficients
• Speaker Modelling by LVQ3 Classifier
• Speaker Recognition based on Segment-Accumulated Distances

Video Processing Module

• Shot Detection

• Face Shot Determination and Mouth Location Estimation

• Mouth Tracking

Audio-Visual Interaction

A combination of the labeling time diagrams ob tained by audio and visual processing is achieved by simple decision rules. Boundaries of a face shot ensure the existence of a person. Mouth movement detection in this shot implies that this person speaks. Non face shot durations cannot be used for speaker detection since interchangeability between speakers cannot be detected by the visual information. Consequently, refinement of the speaker dependent indexing achieved by the audio processing module is performed in face shots with talking faces. The refinement process involves estimation of speaker presence likelihoods in every face shot.

Downloads

-

Relevant Publications

S. Tsekeridou and I. Pitas, "Speaker Dependent Video Indexing based on Audio-Visual Interaction", in Proc. of IEEE Int. Conf. on Image Processing (ICIP'98), vol. 1, pp.358-362, Chicago, Illinois, USA, 4-7 October, 1998.

S. Tsekeridou and I. Pitas, "Speaker Identification for Audio Indexing Applications", in Proc. of IEEE Int. Conf. on Telecommunications (ICT'98), vol. 1, pp. 236-240, Porto Carras, Halkidiki, Greece, 22-25 June, 1998.

Research Projects

NOBLESSE - "Nonlinear Model-Based Analysis and Description of Images for Multimedia Applications", LTR-ESPRIT, EC.

Presentation of the Concealment Techniques

Signal loss occurring in physical communication channels is unavoidable. In the case of transmission of highly compressed video sequences by the MPEG-2 codec, this leads to significant errors observed to the reconstructed frames at the decoder side. These errors fall into two categories:

• Bitstream Errors: they are caused by direct signal loss of some or the whole compressed packet of a coded MB, and result in the loss of the whole respective slice information
• Propagation Errors: they are caused in P- and B-frames uniquely by the additional use of motion compensated time information for their reconstruction at the decoder side. Errors in previously decoded reference frames propagate to the next in the decoding order

The detection of errors in the decoded frames of the MPEG-2 coded and decoded sequence is initially considered. Bitstream errors are indicated by error tokens, sent to the decoder at the time of signal loss occuring. Consequently, the necessary information about the locations of errors (slice and MB) in a frame is already known by the decoder. The problem arises with the detection of the propagation errors. This can be achieved by the use of available space and time information from the current and reference frames at the decoder.

The information about the locations of errors being available, their concealment is successively considered. A different technique is implemented for I-frames, since the latter are coded independently from the other frames of the video sequence. This technique exploits spatial information only from available neighbooring MBs of the current frame.

Error Concealment in P- and B-frames is performed through the use of both space and time information. The space information is obtained by the available neighbooring MBs of the current frame, while time information is acquired by the previously decoded frames.

Performance Results

In order to observe the performance of the concealment techniques, the Flower Garden image sequence has been used and the PSNR of the processed frame has been evaluated. Considering frame 2 (a B-frame) of the mentioned sequence corrupted by both kinds of errors, the error concealment technique, which uses the errorfree frame 0 (I-frame) to obtain the additional time information, gives the following results in respect with PSNR for each chroma component of the frame:

The observation of the performance results leads to the conclusion that the concealment technique performs better, if the previously decoded frames have been very well concealed, in case they were corrupted. The comparison between the errorfree frame 2 and the concealed respective frame proves the validity of this remark.

Error free frame	Corrupted frame	Concealed frame

Downloads

-

Relevant Publications

S.Tsekeridou and I.Pitas, "MPEG-2 Error Concealment based on Block Matching Principles", IEEE Transactions on Circuits and Systems for Video Technology, vol. 10, no. 4, pp. 646-658, June, 2000.

S. Tsekeridou, I. Pitas and C. Le Buhan, "An Error Concealment Scheme for MPEG-2 coded video sequences", in Proc. of 1997 IEEE Int. Symposium on Circuits and Systems (ISCAS'97), Hong Kong, vol.2, pp. 1289-1292, 9-12 June, 1997.

S. Tsekeridou and I. Pitas, "Error Concealment Techniques in MPEG-2", in CD-ROM Proc. of 1997 IEEE Workshop on Nonlinear Signal and Image Processing (NSIP'97), Michigan, USA, 7-11 September, 1997.

Research Projects

-

Schematic describing the feature image sequence modeling by MRBF network

I and J are the location coordinate features assigned to a block site l is the gray level and mx,my are the motion vector components provided by the block matching algorithm

Examples of Moving Object Segmentation and Optical Flow Estimation

Train Sequence Original Frame

Moving Object Segmentation

Optical Flow

Examples of Moving Object Tracking and Prediction

Downloads

Moving Object Tracking Movie (avi file with MJPEG compression)

Relevant Publications

A.G.Bors and I.Pitas, "Optical Flow Estimation and Moving Object Segmentation Based on Median Radial Basis Function Network", IEEE Transactions on Image Processing, vol. 7, no. 5, pp. 693-702, May, 1998.

A. G. Bors and I. Pitas, "Moving Scene Segmentation using Median Radial Basis Function Network", in Proc. of 1997 IEEE Int. Symposium on Circuits and Systems (ISCAS'97), vol. 1, pp. 529-532, Hong Kong, 9-12 June, 1997.

Research Projects

NOBLESSE - "Nonlinear Model-Based Analysis and Description of Images for Multimedia Applications", LTR-ESPRIT, EC.

Presentation of the Adaptive Filters

A usual problem which arises in many image processing applications is the corruption of images by different kinds of noise, which leads to the degradation of their perceived quality. To deal with this problem, researchers in the field of image processing and analysis, have developed, over the years, various filtering algorithms for noise removal. Three adaptive nonlinear order statistics filters for noise removal are:

• Adaptive LMS L Filter: the output is defined by the linear combination of the order statistics of the input samples in the filter window

  

  where the coefficient vector a (k) is adapted at each step k accordingly to the LMS adaptation algorithm.

• Adaptive LMS Ll Filter: it is an extension of the adaptive LMS L filter in that it preserves both space or time and order information. This is achieved by proper modification of the ordered input vector. Its output is calculated by use of the equation
  

  For the adaptation of its coefficient vector c (k), the LMS algorithm is considered here as well.

• Modified Signal Adaptive (SAM) Median Filter: they adapt their behaviour in accordance with the local signal to noise ratio. Thus, they behave differently in homogeneous regions or edge regions. Their output is given by:
  

  According to the method of window adaptation employed, the modified signal adaptive median filter is further distinguished to the:

• Symmetrical Signal Adaptive Median Filter: the window size is adapted in a symmetrical way
• Morphological Signal Adaptive Median Filter: assymetrical window increment/decrement is performed

Performance Results

For the comparison of the performance of the proposed filters, a reference image called has been corrupted by the contaminated gaussian noise model:

The performance results in respect with SNR, PSNR, MAE and MSE, measured on the processed versions of a noisy image by the adaptive L, by the adaptive Ll, by the symmetrical SAM and by the morphological SAM filters, are concentrated in the following table:

The observation of both performance results and processed images leads to the following conclusions:

• The adaptive Ll filter exhibits better performance results, for medium corrupted images, while the SAM filters are better in high corruption cases.
• Considering the subjective criterion of perceived image quality of the processed image, the SAM filters perform better.

Original Image		Noisy Image
Adaptive LMS L Noise Removal		Adaptive LMS LI Noise Removal
Symmetrical SAM Noise Removal		Morphological SAM Adaptive LMS LI Noise Removal

Downloads

-

Relevant Publications

S. Tsekeridou, C. Kotropoulos and I. Pitas, "Adaptive Order Statistic Filters for the Removal of Noise from Corrupted Images", SPIE Optical Engineering, vol. 37, no. 10, pp. 2798-2816, October, 1998.

C. Kotropoulos and I. Pitas, "Adaptive LMS L-filters for Noise Suppression in Images", IEEE Transactions on Image Processing, vol. 5, no. 12, pp. 1596-1609, December, 1996.

S. Tsekeridou, C. Kotropoulos and I. Pitas, "Morphological Signal Adaptive Median Filter for Noise Removal", in Proc. of 1996 Int. Conf. on Electronics, Circuits and Systems (ICECS'96), vol. 1, pp. 191-194, Rodos, Greece, 13-16 October, 1996.

Research Projects

-

Downloads

-

Relevant Publications

V.Chatzis and I.Pitas, "Fuzzy Cell Hough Transform for Curve Detection", Pattern Recognition, Elsevier, vol. 30, no. 12, pp. 2031-2042, December, 1997.

V. Chatzis and I. Pitas, "Randomized Fuzzy Cell Hough Transform",in Proc. of the 6th IEEE Int. Conf. on Fuzzy Systems (FUZZ-IEEE'97), vol. 2, pp. 1185-1190, Barcelona, Spain, 1-5 July, 1997.

V. Chatzis and I. Pitas, "Introducing the Select and Split Fuzzy Cell Hough Transform", in Proc. of ICPR '96, vol. 2, pp. B552-B556, Vienna, Austria, 25-30 August, 1996.

V. Chatzis and I. Pitas, "Select and Split Fuzzy Cell Hough Transform-A fast and efficient method to detect contours in images", in Proc. of the Fifth IEEE Int. Conf. on Fuzzy Systems (FUZZ-IEEE'96), vol. 3, pp. 1892-1898, New Orleans, USA, 8-11 September, 1996.

V. Chatzis and I. Pitas, "Fuzzy Cell Hough Transform", in Proc. of EUSIPCO-96, vol. 3, pp. 1717-1720, Trieste, Italy, September, 1996.

Research Projects

NOBLESSE - "Nonlinear Model-Based Analysis and Description of Images for Multimedia Applications", LTR-ESPRIT, EC.

NAT - "Nonlinear and adaptive digital image processing and computer vision"