See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228806073

Real-Time Video Foreground Extraction Based on Context-Aware Background

Substraction

Article · January 2007

CITATIONS READS

2 172

2 authors:

Alvaro Garcia-Martin Jesús Bescós

Universidad Autónoma de Madrid Universidad Autónoma de Madrid
34 PUBLICATIONS   662 CITATIONS    58 PUBLICATIONS   385 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

VISION View project

Tracking Object View project

All content following this page was uploaded by Alvaro Garcia-Martin on 25 March 2015.

The user has requested enhancement of the downloaded file.

 REAL TIME VIDEO FOREGROUND EXTRACTION BASED ON CONTEXT-AWARE
BACKGROUND SUBTRACTION*

Alvaro García, Jesús Bescós

Grupo de Tratamiento de Imágenes, Escuela Politécnica Superior

Universidad Autónoma de Madrid, E-28049 Madrid, Spain
e-mail: {Alvaro.Garcia, J.Bescos}@uam.es

ABSTRACT background update stage. However, it is not used to

improve directly the segmentation performed by
This paper describes a real-time approach for background subtraction. This results in a slight
foreground segmentation in image sequences taken improvement in the subtraction phase. As explained in
from a stationary camera, as a first preprocessing step the next section, temporal change can also be exploited
for object extraction. The proposed method combines a in the foreground/background classification stage of the
background subtraction method with the extensible background subtraction method.
inclusion of context information. It currently considers In [10] a novel method for moving object detection
bidirectional temporal change detection as the a priori is proposed, combining bidirectional frame difference
context data (i.e., desired objects are moving). This with background difference. It achieves good results
combination has proven to improve the detection of the with highly textured objects or when there are high
object boundaries, hence enhancing the reliability of the differences between consecutive frames (e.g., for low
segmentation masks, with little extra computational frame rates). When applied to typical sequences, where
load. The tested implementation of the proposed objects can be low textured and there is little change
foreground detector shows a good trade-off between between consecutive frames, homogeneous regions
efficiency, segmentation accuracy and robustness, inside objects are incorrectly classified as background.
particularly compared with some other state-of-the-art However, areas belonging to the edges of moving
methods. objects (where changes are more obvious) are still well
classified. The thickness of these borders increases
Index Terms— video analysis, background subtraction, when the objects position changes significantly from
change detection, context information. one frame to the next. Foreground objects detected by
this method are usually not completely detected.
1. INTRODUCTION However, it seems a good starting point for more
sophisticated methods, especially taking into account
Foreground detection, or background extraction, is a the fact that most of the background subtraction
very common preprocessing step for many higher-level approaches reduce the quality of their segmentation at
applications in computer vision. low frame rates, frequently used for high resolution
The majority of methods in the literature rely on real-time operation.
variations of two basic approaches: frame difference Our goal is to classify, in real time, points on the
evaluation and background subtraction. Some of them image not belonging to the background. which for the
make use of both, trying to balance between their targeted context is assumed to be non-complex. This
advantages and drawbacks. means that there are no moving elements in the
For most surveillance applications, the camera is background. As this segmentation will be later used for
assumed to be stationary. In this context, methods based object detection and tracking, we will give special
on background subtraction have obtained the best importance to object boundaries in the foreground
results. Many different techniques can be applied to segmentation, as they will define object shapes.
classify foreground pixels and then construct a This paper presents a method that combines a
background model, like a mixture-of-gaussians model context driven model of the foreground to improve its
(GMM)[1][2][3], eigen-backgrounds[4], or mean-shift detection via background subtraction. The presented
based estimation[5][6]. A brief review on these and approach applies bidirectional temporal change
other popular techniques can be found in[7]. * detection (i.e., desired objects are moving) to achieve it.
In order to improve segmentation, other techniques This paper is structured as follows: section 2
are combined with background subtraction[8][9]. In [8], describes the proposed approach; section 3 presents a
a running gaussian average is combined with temporal working application, and experimental results; finally,
change detection in order to prevent errors in the section 4 draws the conclusions.

*
Work partially supported by the Spanish Government
under Project TIN2004-07860-C02-01 (Medusa).
 2. ALGORITHM DESCRIPTION
This method is efficient and shows little false
This method is designed to work with different positives (background points classified as foreground),
point sizes, which can be either pixels or pixel blocks. but fails in homogeneous (low-textured) areas of both
Election of the size of these points will depend on the the foreground and the non-moving objects, where
requirements of the application. Grouping pixels in points are very similar between consecutive frames, and
blocks will provide greater efficiency and robustness frame difference is not able to robustly detect changes.
against noise, although obtained mask will not have In order to improve robustness against noise and
pixel-accuracy. Working with pixels will provide a finer parameter values, we perform background subtraction
segmentation, with loss of efficiency and robustness. with a single-gaussian model, instead of the fixed
The working flow of the full method is illustrated thresholding method used in [10]. The appearance of
in Figure 1. The first stage performs a temporal change speckles in classified areas usually associated with
detection, which will try to detect the moving gaussians in the background model is prevented by the
foreground. This first segmentation is detailed in combination with frame difference masks.
subsection 2.1. The mask obtained from this first stage
will be used as context information in the second stage. 2.2. Context-aware background subtraction
The second stage consists of a context-aware
background subtraction algorithm, which will yield the Assuming a non-complex background in the input
final segmentation. It is a modified running gaussian sequences (static camera, no moving elements in the
average algorithm that takes into account not only the background), a single gaussian model for each point is
background model and the incoming frames, but also sufficient. As explained in [7], a mixture of gaussians
external context information, currently representing a algorithm with a single gaussian in its model is
priori confidence about moving objects. This algorithm equivalent to a running average gaussian. In this way,
will be described in section 2.2. we are modelling the averaged value of each point
. (mean value) and an estimation of its noise over time
(the standard deviation).
Context information aims to provide confidence on
each point’s belonging to the foreground. In this sense,
it conforms an a priori confidence mask, represented by
Mconf. Currently, this mask results from the
segmentation performed in the temporal change
detection stage (i.e., Mconf=Mtc) . However, we are
testing the inclusion of other context based criteria in
the same framework. Luminance and texture
homogeneity, object connectivity and compactness, or
coherence of the object motion (extracted via a tracking
algorithm [9]), could be easily integrated into the
system, providing improvement in robustness and
sensitivity.
Our algorithm is based on the one described in [8],
modified in two ways in order to account for context
information both in the classification and in the
background model updating phases, as explained in the
following subsections..

2.2.1. Variable weight background/foreground

Figure 1: Overview of the implemented method classification
The first modification is the use of Mconf in the
background/foreground classification process. In the
2.1. Temporal change detection classic running gaussian average[7], a point value in It
is classified as foreground if the inequality:
This stage is based on the method proposed in [10].
Temporal change is explored in both directions. The |It – μt| > k·σt
previous and posterior frames are compared to the
considered frame obtaining two frame difference masks holds, being μt and σt the mean and standard deviation
(called Mfd-prev and Mfd-post) and a rough background for that point in the background model, and k a fixed
subtraction is also performed (Mbg_sub). Obtained masks value. In our approach, we propose k to be dependent of
are combined in order to get an initial segmentation the value of Mconf for each point. The higher the
mask : confidence in a point’s belonging to foreground, the
lower the value of k for that point.
Mtc = (Mfd-prev AND Mbg_sub) OR (Mfd-post AND Mbg_sub)
 As we currently use a binary version of Mconf (it
represents presence or absence of motion), k will take 3. EXPERIMENTS AND RESULTS
one of two possible values, kmin for changed points and
kmax for non-changed points. The described implementation has been tested into a
The result of this classification will provide the working system: the Medusa project. The goal of one of
final background/foreground segmentation mask (Mfore). its applications is to count people crossing an inter-
building corridor with glass walls to the exterior. This
2.2.2. Robust background model adaptation scenario provides a non-complex background with high
Currently, methods based on background difference degree of illumination changes, shadows and
update their background model only in those points reflections.
classified as background. This prevents foreground This application has been tested using different
objects from corrupting the background model. Tjis point sizes: 8x8 blocks and pixels. Working with 8x8
motivates our second modification to the basic blocks greatly improves efficiency at the expenses of
algorithm. We combine the received contextual lowering shape resolution, but this might be enough for
information with the final background/segmentation some applications as the one we show. Moreover, this
mask before using it for selective background update. In could act as a rough confidence mask for a pixel level
the current implementation, as contextual information is segmentation.
represented by a binary mask, they are just or’ed to Regarding to computational efficiency, the
obtain the background model update mask (Mbkg_upd). application has been implemented in C++, using the
To ensure a clean background model, only points with a OpenCV[12] library for some image processing
high degree of confidence about its belonging to the operations. Tests have been executed on a general
background (i.e. inside Mbkg_upd) will be updated in the purpose PC (PIV, 3.0 GHz, 1GB-RAM), which is able
background model, preventing typical foreground to simultaneously run the application over sequences
classification errors from polluting the background from three cameras with a resolution of 640x480 and 15
model. fps. The results can be checked online in the Medusa
This proposed method includes into the foreground project web page: http://dymas.ii.uam.es/~agm/.
concept all objects that are not initially in the Compared to techniques just based on running
background: moving objects, shadows, reflections, gaussian average[7][8], foreground objects detected by
stopped objects that were not initially into the our combined method have their borders more defined.
background and background regions initially occluded So, misclassified holes inside these objects can be
by static background objects that began to move. This detected as real holes and included into the foreground.
approach reduces the risk of foreground objects being The combined method has also shown greater
blindly included into the background, and allows for an robustness against image noise.
object detection system that can later distinguish In order to comparatively evaluate overall
between them, applying its own knowledge, based on performance of the proposed approach, two other state-
context information, of the kind of objects that can be of-the-art methods have also been tested into the
present in the foreground. aforementioned application. The first one uses an
Assuming a non-complex background, common in improved GMM[3], and the second a statistical
surveillance applications, static objects initially detected approach[11] (as implemented in the OpenCV library).
as foreground would not be blindly included into the Our results both for live video and for specifically
background as time passes by. Instead, a maximum time generated test sequences, show that the proposed
is set for a static object to be consciously included into method outperforms them in several aspects: quality of
the background model. segmentation at low frame rates, initialization time and

(a) (b) (c) (d) (e)

(a) (b) (c) (d) (e)

Figure 2: Segmentation results obtained for two frames in one of the test sequences.: (a) Current frame; (b) proposed
method, 8x8 blocks; (c) proposed method, pixels; (d) improved GMM algorithm [3]; (e) statistical approach [11].
 efficiency. Our algorithm has proven to obtain precise Proceedings Image and Vision Computing New Zealand,
segmentation masks for high and low frame rates 2002.
without modifying the value of its parameters (however,
significant changes in the noise level require the [3] P. KaewTraKulPong, R. Bowden, “An Improved
Adaptive Background Mixture Model for Real-time Tracking
adjustment of some of them). Figure 2 shows qualitative with Shadow Detection”, Proc. 2nd European Workshop on
results of the achieved segmentation, while Table 1 Advanced Video Based Surveillance Systems, AVBS01. Sept
shows measured efficiency for each tested method. 2001.

Image size 352x288 640x480 [4] J. Rymel, J. Renno, D. Greenhill, J. Orwell, G.A. Jones,
Proposed method (pixels) 62 fps 20 fps "Adaptive eigen-backgrounds for object detection," Image
Proposed method (8x8 blocks) >300 fps 95 fps Processing, 2004. ICIP '04. 2004 International Conference on
GMM[3] 25 fps 8 fps , vol.3, no.pp. 1847- 1850 Vol. 3, 24-27 Oct. 2004
Statistical approach[11] 20 fps 5 fps
[5] M. Piccardi, T. Jan, "Mean-shift background image
Table 1: Efficiency for tested algorithms. modelling," Image Processing, 2004. ICIP '04. 2004
International Conference on , vol.5, no.pp. 3399- 3402 Vol. 5,
Typical values for execution parameters are shown in 24-27 Oct. 2004
Table 2: Ttc, is the threshold for frame difference, τacc
the length of the adaptation window for background [6] B. Han, D. Comaniciu, L. Davis, “Sequential kernel
subtraction operations[8], k0 the value for the density approximation through mode propagation:
confidence factor in the background subtraction applications to background modeling”, Proc. ACCV – Asian
described in section 2.1, and kmax and kmin are explained Conf. on Computer Vision, 2004.
in section 2.2.
[7] M. Piccardi, "Background subtraction techniques: a
Symbol Value Meaning review," Systems, Man and Cybernetics, 2004 IEEE
Ttc 15-30 Temporal change threshold International Conference on , vol.4, no.pp. 3099- 3104 vol.4,
10-13 Oct. 2004.
τacc 100-200 Length of bkg adaptation window
k0 2.5 Blind confidence factor
[8] S. Huwer, H. Niemann, "Adaptive Change Detection for
kmax 3-6 Non-changed point confidence factor Real-Time Surveillance Applications" Visual Surveillance,
kmin 1-2 Changed point confidence factor 2000. Proceedings. Third IEEE International Workshop on,
Table 2: Parameter values for a typical indoor application. vol., no.pp.37-46, 2000.

[9] V. Mezaris, I. Kompatsiaris, N. Boulgouris, M. Strintzis,

4. CONCLUSIONS “Real-Time Compressed-Domain Spatiotemporal
Segmentation and Ontologies for Video Indexing and
This paper proposes a method for foreground detection Retrieval”, IEEE Trans. Circuits Syst. Video Technol., May
2004
in video sequences, using temporal change detection to
improve the results of a background subtraction [10] S.M. Desa, Q.A. Salih, "Image Subtraction for Real Time
method. Assuming a non-complex background, Moving Object Extraction," Computer Graphics, Imaging and
common in many indoor surveillance applications, the Visualization, 2004. CGIV 2004. Proceedings. International
method is able to work in real-time while adapting to Conference on , vol., no.pp. 41- 45, 26-29 July 2004.
smooth illumination changes.
Results show a good compromise between [11] L. Li, W. Huang, I.Y.H. Gu, Q. Tia, “Foreground object
efficiency, accurate segmentation and robustness, being detection from videos containing complex background”,
able to work with high and low frame rates while Proceedings of the eleventh ACM international conference on
Multimedia, November 02-08, 2003, Berkeley, CA, USA.
complying with real-time requirements.
This method introduces a context-aware background [12] OpenCV, open source library for computer vision.
subtraction method as its main innovation. In the http://www.intel.com/technology/computing/opencv/overview.
current implementation, temporal change detection htm
represents this context information. However, as
pointed out, the framework allows further use of
additional context information to improve the
background subtraction with very little effort.

REFERENCES

[1] C. Stauffer, W.E.L. Grimson, "Adaptive Background

Mixture Models for Real-Time Tracking," CVPR 1999, p.
2246, 1999.

[2] P.W. Power, J.A. Schoonees, “Understanding

Background Mixture Models for Foreground Segmentation”,

View publication stats