Hierarchical Video Database Indexing and Access Control

  • Proposed Problems: Digital video now plays an important role in education, healthcare, entertainment and other multimedia applications. Several content-based video retrieval (CBVR) systems have been introduced in the past, and they have advanced our capabilities for searching videos via color, layout, texture, motion and shape features. The performance of these existing CBVR systems would be greatly enhanced if we can build suitable hierarchical database indexing and access control techniques. However, hierarchical video database indexing and content-based access control are still challenging and open problems because of: (a) Semantic Gap: There is still no widely accepted approach to overcome the semantic gap between low-level visual features and high-level semantic visual concepts. (b) Relation Gap: When very large video data set comes into view, efficient video database indexing can no longer be ignored. However, conventional database indexing trees cannot be used for video database indexing because of the curse of dimensions and the semantic gap. This problem reflects a gap between two traditional independent research fields: the field of databases and the field of computer vision and image processing. Thus, we call the problem relation gap for brevity. (c) Access Control Problem: The lack of access control mechanisms is another common weakness of the existing video retrieval systems. The development of such mechanisms is increasingly relevant because video data today are used for different purposes. Content-adaptive and user-dependent video database access control is becoming one of the emerging needs.
  • Proposed Research: The proposed research will introduce a novel and unified framework to tackle the three problems mentioned above. Specifically, the framework will include: (a) A novel content-based video analysis technique to obtain the suitable video content representation and feature extraction framework. (b) A hierarchical database indexing structure to enable more effective video access over large-scale collections. (c) A hierarchical video summarization technique to enable concept-oriented browsing of large-scale video collections. (d) A rule-based video classifier to relate the low-level visual features to the high-level semantic visual concepts. (e) A unified video database access framework to enable the naive users specifying their query concepts easily and effectively.

    • Project Investigators:

  • Jianping Fan (PI), UNC-Charlotte;
  • Jing Xiao (Co-PI), UNC-Charlotte;

    • Medical Consultant:

  • Dr. James F. Kellam, Carolinas Medical Center;

    • PhD Students:

  • Hangzai Luo;
  • Yuli Gao;

    • Project Sponsors:

  • National Science Foundation: IIS0208539;
  • AO Foundation, Switzerland;

    • Project Reports:

  • J. Fan, J. Xiao, First Year NSF Report, April, 21, 2003.
  • J. Fan, J. Xiao, Second Year NSF Report, April, 21, 2004.
  • J. Fan, J. Xiao, Third Year NSF Report, July, 7, 2005.
  • J. Fan, J. Xiao, NSF Final Report, August 18, 2005.

    Current Project Achievements

    a. System Implementation:

  • System Demo

    First, download this zip file; Second, generate a folder and unzip the file and save them in the same folder; Third, double click Player.exe
  • Salient Object Detection and Tracking:
  • Semantic Video Classification:
  • Concept-Oriented Video Summarization and Skimming:
  • Applications for Online Medical Education:

    • b. Journal Publications:

  • J. Fan, H. Luo, E. Bertino, ``Constructing distributed Hippocratic video databases for privacy-preserving online patient training and counseling", ACM Trans. on Information Systems, 2005 (accepted).
  • H. Luo, X. Xue, J. Fan, ``Hierarchical classification of surgery education videos to enable concept-oriented video summarization and skimming", ACM Trans. on Multimedia Computing, Communications, and Applications, 2005 (accepted).
  • J. Fan, Y. Gao, H. Luo, G. Xu, ``Statistical modeling and conceptualization of natural images", Pattern Recognition, vol.38, pp.865-885, 2005.
  • J. Fan, G. Zeng, M. Body, M.-S. Hacid, ``Seeded region growing: an extensive and comparative study", Pattern Recognition Letters, vol.26, pp.1139-1156, 2005.
  • X. Zhu, J. Fan, X. Wu, W. Aref, A.K. Elmagarmid, ``Exploiting Video Content Structure for Hierarchical Summarization'', ACM Multimedia Systems, vol.10, no.2, pp.98-115, 2004.
  • W. Aref, A. Catlin, A. Elmagarmid, J. Fan, M. Hammad, I. Ilyas, M. Marzouk, S. Prabhakar, X. Zhu, "A testbed facility for research in video database benchmarking ", ACM Multimedia Systems, Special Issue on Multimedia Document Management Systems, vol.9, no.6, pp.575-585, 2004.
  • J. Fan , H. Luo, A.K. Elmagarmid, ``Concept-Oriented Indexing of Video Database: Towards More Effective Retrieval and Browsing", IEEE Trans. on Image Processing, vol.13, no.5, 2004.
  • J. Fan , X. Zhu, A.K. Elmagarmid, W.G. Aref, L. Wu, ``ClassView: Hierarchical Video Shot Classification, Indexing, and Accessing", IEEE Trans. on Multimedia, vol.6, no.1, pp.70-87, 2004. The demo was present on ICDE 2002, San Jose, CA.
  • E. Bertino, J. Fan, E. Ferrari, M.-S. Hacid, A.K. Elmagarmid, Xingquan Zhu, ``A Hierarchical Access Control Model for Video Database Systems", ACM Trans. on Information Systems, vol.21, no.2, pp.155-191, 2003.
  • J. Fan, X. Zhu, L. Wu, ``Accessing video contents through key objects over IP", Multimedia Tools and Applications, vol.21, pp.75-95, 2003.
  • X. Zhu, J. Fan , A.K. Elmagarmid, ``Hierarchical visual summarization and content description by using joint similarity with semantics and visual perception", ACM Multimedia Systems, vol.9, no.1, pp.31-53, 2003.
  • J. Fan, Mohand-Said Hacid, Feng Liang, ``Novel Tracking-Based Moving Object Extraction Algorithm", Journal of Electric Imaging, vol.11, no.3, pp.393-403, 2002.
  • J. Fan, X. Zhu, Mohand-Said Hacid, Ahmed K. Elmagarmid, ``Model-based video classification toward hierarchical representation, indexing and access" , Multimedia Tools and Applications, vol.17, pp.97-120, 2002.

    • c. Conference Publications:

  • J. Fan, H. Luo, M.-S. Hacid, E. Bertino, ``A novel approach to enable privacy preserving video sharing", ACM CIKM Conference, Bremen, Germany, 2005.
  • J. Fan, H. Luo, Y. Gao, M.-S. Hacid, ``Mining image database on semantic via statistical learning", ACM SIGKDD Conference, Chicago, 2005.
  • J. Fan, H. Luo, Y. Gao, ``Learning the semantics of images by using unlabeled samples", IEEE CVPR Conference, San Diego, 2005.
  • J. Fan, Y. Gao, H. Luo, ``Multi-level annotation of natural scenes using dominant image components and semantic concepts", ACM Multimedia Conference, New York, pp.540-548, 2004.
  • H. Luo, J. Fan, ``Concept-oriented video skimming via semantic video classification", ACM Multimedia Conference, New York, pp.760-763, 2004.
  • H. Luo, J. Fan, ``Concept-Oriented video skimming and adaptation via semantic classification", ACM Multimedia Workshop on Multimedia Information Retrieval, New York, 2004.
  • J. Fan, Y. Gao, H. Luo, G. Xu, ``Semantic Video Classification and Feature Subset Selection by Using Unlabeled Samples'', ACM SIGIR, Sheffield, UK, 2004.
  • J. Fan, Y. Gao, H. Luo, G. Xu, ``Automatic Image Annotation by Using Concept-Sensitive Salient Objects for Image Content Representation'', ACM SIGIR, Sheffield, UK, 2004.
  • J. Fan, H. Luo, J. Xiao, L. Wu, ``Semantic Video Classification and Feature Subset Selection under context and concept uncertainty'', ACM JCDL, Tucson, USA, 2004.
  • Y. Gao, J. Fan, H. Luo, G. Xu, ``Salient Objects: Semantic Building Blocks for Image Concept Interpretation'', CIVR, 2004.
  • H. Luo, J. Fan, Y. Gao, G. Xu, ``Multimodal Salient Objects: General Building Blocks of Semantic Video Concepts'', CIVR, 2004.
  • J. Fan, H. Luo, L. Wu, ``Integrating Unlabeled Samples for Semantic Video Classification and Feature Subset Selection", ACCV (Asian Conf. on Computer Vision), Jan. 27-30, 2004.
  • H. Luo, Y,. Gao, J. Fan, ``Semantic Video Classification with Insufficient Labeled Samples", SPIE: Storage and Retrieval of Media Database, San Jose, CA, Jan. 18-22, 2004.
  • H. Luo, Y,. Gao, J. Fan , ``Principal Video Shot and Physical Video Shots: Which one is better for semantic video classification", SPIE: Storage and Retrieval of Media Database, San Jose, CA, 2004.
  • J. Fan , H. Luo, ``Principal Video Shot: Linking low-level multimodal perceptual features to semantic video events:, IEEE CVPR Workshop on Event Mining, Madison, WI, 2003.
  • J. Fan , H. Luo, ``Semantic Medical video classification by integrating adaptive EM algorithm with flexible mixture model", ACM Multimedia Workshop on Multimedia Information Retrieval, Berkeley, CA, 2003.
  • H. Luo, J. Fan, J. Xiao, X. Zhu, ``Multimodal principal video shot classification via mixture Gaussian", Proc. ICME, special session on Moving from Features to Semantics, pp.189-192, 2003.
  • J. Fan, Y. Gao, H. Luo, M.-S. Hacid, ``A novel framework for semantic image classification and benchmark", ACM SIGKDD Workshop on Multimedia Data Mining, 2003.
  • W. Aref, A.K. Elmagarmid, J. Fan, M. Hammad, I. Ilyas, M. Marzouk, S. Prabhakar, Y. Tu, X. Zhu, ``VDBMS: A testbed facility for research in video database benchmark", ( invited paper), 9th International Conference on Distributed Multimedia Systems, Miami, Sept. 23-25, 2003.
  • J. Fan, H. Luo and X. Zhu, ``Semantic principal video shot classification system", ACM Multimedia Workshop on Multimedia Information Retrieval, Juan Les Pins, France, Dec., 2002.
  • X. Zhu, J. Fan, H. Luo and M.-S. Hacid, ``Using small samples for content-based image retrieval system with relevance feedback", ACM Multimedia Workshop on Multimedia Information Retrieval, Juan Les Pins, France, Dec., 2002.
  • X. Zhu, W.G. Aref, J. Fan, A.C. Catlin, A.K. Elmagarmid, ``Medical Video Mining for Efficient Database Indexing, Management and Access", International Conference on Data Engineering (ICDE'03), Bangalore, India, March 5-March 8, 2003.
  • J. Fan, M. Body, X. Zhu, M.-S. Hacid, ``Seeded image segmentation towards content-based image retrieval", Proc. SPIE: Storage and Retrieval of Media Database, vol.4676, pp.10-21, 2002.
  • X. Zhu, J. Fan, W.G. Aref, A.K. Elmagarmid, ``Hierarchical video summarization for medical videos" , Proc. SPIE: Storage and Retrieval of Media Database, vol.4676, pp.395-406, 2002.
  • X. Zhu, J. Fan, A.K. Elmagarmid, W.G. Aref, ``ClassMiner: Hierarchical video event mining for medical videos", ACM SIGMOD Workshop on Data Mining and Knowledge Discovery, Madison, 2002.
  • X. Zhu, J. Fan, A.K. Elmagarmid, W.G. Aref, ``Facial feature location and verification in image and videos", Proc. ICIP, 2002.
  • X. Zhu, J. Fan, X. Xue, L.Wu, A.K. Elmagarmid, ``Semi-automatic video annotation", Third IEEE Pacific Rim Conference on Multimedia (PCM'02), 2002.
  • X. Zhu, X. Xue, J. Fan, L.Wu, ``Qualitative camera motion classification for content-based video indexing", Third IEEE Pacific Rim Conference on Multimedia (PCM'02), 2002.
  • W.G. Aref, A. Catlin, J. Fan, A.K. Elmagarmid, M. Hammad, I. Ilyas, M. Marzouk, ``A video database management system for advancing video database research", International Workshop on Multimedia Information Systems, Tempe, Arizona, USA, Oct.30-Nov.1, 2002.

    • d. Book Chapters:

  • J. Fan, X. Zhu, X. Lin, ``Video Data Mining", in Multimedia Data Mining, edited by Dr. Chabane Djeraba, Kluwer Academic Publishers, 2002.
  • J. Fan, X. Zhu, J. Xiao, ``Content-Based Video Indexing and Retrieval", in Computer Graphics and Multimedia: Applications, Problems and Solutions, edited by Prof. John DiMarco, Ideas Publishers, 2003.

    • e. Demo Shows on the Leading Conferences:

  • 18th IEEE International Conferences on Data Engineering (ICDE), San Jose, CA, Jan., 2002: "A Distributed Database Server for Continuous Media" .
  • ACM Multimedia, Juan-les-Pins, France, Dec.1-6, 2002: "ClassMiner: Mining Medical Videos for Scalable Skimming and Summarization".
  • ACM Multimedia, New York, 2004: ``Concept-oriented video skimming via semantic video classification".

    • Project References

    1. Related Journals and Conferences:

  • IEEE Trans. on Image Processing;
  • IEEE Trans. on Multimedia;
  • IEEE Trans. on Pattern Analysis and Machine Intelligence;
  • IEEE Trans. on Circuits and Systems for Video Technology;
  • ACM Trans. on Information Systems;
  • Journal of Electronic Imaging;
  • ACM Multimedia Systems;
  • Multimedia Tools and Applications;
  • ACM Multimedia Conference;
  • IEEE ICME, IEEE CVPR, IEEE PR, IEEE ICCV, IEEE ICIP;
  • SPIE Conference on Storage and Retrieval for Media Databases;

    • 2. Area References:

  • C. Faloutsos, et al., ``Efficient and effective querying by image content", Journal of Intelligent Information Systems, vol.3, pp.231-262, 1994.
  • A. Pentland, R. Picard, S. Sclaroff, ``Photobook: content-based manipulation of image databases", International Journal of Computer Vision, vol.18, 1996.
  • J.D. Courtney, ``Automatic video indexing via object motion analysis", Pattern Recognition, vol.30, pp.607-626, 1997.
  • S.F. Chang, et al., ``A fully automatic content-based video search engine supporting spatiotemporal queries", IEEE Trans. on Circuits and Systems for Video Technology, vol.8, pp.602-615, 1998.
  • Y. DEng, B.S. Manjunath, ``Netra-V: Toward an object-based video representation", IEEE Trans. on Circuits and Systems for Video Technology, vol.8, pp.616-627, 1998.
  • S. Satoh, T. Kanade, ``Name-It: Association of face and name in video", CVPR, 1997.
  • J. Fan, et al., ``MultiView: Multi-level video content representation and retrieval", Journal of Electronic Imaging, vol.10, pp.895-908, 2001.
  • M. Naphade, T.S. Huang, ``A probabilistic framework for semantic video indexing, filtering, and retrieval", IEEE Trans. on Multimedia, vol.3, pp.141-151, 2001.
  • H.J. Zhang, et al., ``Automatic partitioning of full-motion video", Multimedia Systems, 1993.
  • E. Bertino, J. Fan, E. Ferrari, M.-S. Hacid, A.K. Elmagarmid, Xingquan Zhu, ``A Hierarchical Access Control Model for Video Database Systems", ACM Trans. on Information Systems, vol.21, no.2, pp.155-191, 2003.
  • Y. Day, A.A. Khokhar, S. Dagtas, A. Ghafoor, ``A multi-level abstraction and modeling in video databases'', {\em Multimedia Systems}, vol.7, pp.409-423, 1999.

    • 3. Related Projects:

  • Informedia Project at CMU.
  • Advent Project at Columbia University.

    • Remarks:

  • We are still working on this project, we will update our results in the future!

    • If we know what we were doing, it wouldn't be research, would it? ---Albert Einstein(1879-1955)---