[go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

F2ConText: how to extract holistic contexts of persons of interest for enhancing exploratory analysis

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

A wide variety of publicly available heterogeneous data has provided us with an opportunity to meander through contextual snippets relevant to a particular event or persons of interest. One example of a heterogeneous source is online news articles where both images and text descriptions may co-exist in documents. Many of the images in a news article may contain faces of people. Names of many of the faces may not appear in the text. An expert on the topic may be able to identify people in images or at least recognize the context of the faces who are not widely known. However, it is difficult as well as expensive to employ topic experts of news topics to label every face of a massive news archive. In this paper, we describe an approach named F2ConText that helps analysts build contextual information, e.g., named entity context and geographical context of facial images found within news articles. Our approach extracts facial features of the faces detected in the images of publicly available news articles and learns probabilistic mappings between the features and the contents of the articles in an unsupervised manner. Afterward, it translates the mappings to geographical distributions and generates a contextual template for every face detected in the collection. This paper demonstrates three empirical studies—related to construction of context-based genealogy of events, tracking of a contextual phenomenon over time, and creation of contextual clusters of faces—to evaluate the effectiveness of the generated contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. Codes and data are provided here: http://dal.cs.utep.edu/projects/storyboarding/KAIS/. Password: 16context.

References

  1. Ahonen T, Matas J, He C, Pietikäinen M (2009) Rotation invariant image description with local binary pattern histogram Fourier features. In: SCIA

  2. Alias-i (2011) LingPipe 4.1.0. http://alias-i.com/lingpipe/. Accessed 30 Sept 2018

  3. Bier EA, Ishak EW, Chi E (2006) Entity workspace: an evidence file that aids memory, inference, and reading. In: ISI, pp 466–472

  4. Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. JMLR 3:993–1022. http://dl.acm.org/citation.cfm?id=944919.944937

  5. Bruni E, Tran GB, Baroni M (2011) Distributional semantics from text and images. In: Proceedings of the GEMS 2011 workshop on geometrical models of natural language semantics. Association for Computational Linguistics, pp 22–32

  6. Bruni E, Tran N-K, Baroni M (2014) Multimodal distributional semantics. J Artif Intell Res 49:1–47

    Article  MathSciNet  MATH  Google Scholar 

  7. Cai D, Yu S, Wen J-R, Ma W-Y (2003) VIPS: a vision-based page segmentation algorithm. Technical report, MSR-TR-2003-79

  8. Chakrabarti S, Sarawagi S, Dom B (1998) Mining surprising patterns using temporal description length. In: VLDB ’98, vol 98, pp 606–617

  9. Chen L, Hu B, Zhang L, Li M, Zhang H (2003) Face annotation for family photo album management. IJIG 3(01):81–94

    Google Scholar 

  10. Choi JY, De Neve W, Ro YM, Plataniotis KN (2010) Automatic face annotation in personal photo collections using context-based unsupervised clustering and face information fusion. IEEE Trans Circuits Syst Video Technol 20(10):1292–1309

    Article  Google Scholar 

  11. Choi JY, Yang S, Ro YM, Plataniotis KN (2008) Face annotation for personal photos using context-assisted face recognition. In: Proceedings of the 1st ACM international conference on multimedia information retrieval. ACM, New York, pp 44–51

  12. Christopher D, Manning PR, Schtze H (2008) Introduction to information retrieval, chapter 13. Text classification and Naive Bayes. Cambridge University Press, Cambridge

    Google Scholar 

  13. Das-Neves F, Fox EA, Yu X (2005) Connecting topics in document collections with stepping stones and pathways. In: CIKM ’05, pp 91–98

  14. Ester M, peter Kriegel HSJ, Xu X (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD

  15. Faloutsos C, McCurley KS, Tomkins A (2004) Fast discovery of connection subgraphs. In: KDD 04’, pp 118–127

  16. Fang L, Sarma AD, Yu C, Bohannon P (2011) Rex: explaining relationships between entity pairs. Proc VLDB Endow 5(3):241–252

    Article  Google Scholar 

  17. Fauzi F, Hong J-L, Belkhatir M (2009) Webpage segmentation for extracting images and their surrounding contextual information. In: MM. https://doi.org/10.1145/1631272.1631379

  18. Feng S, Manmatha R, Lavrenko V (2004) Multiple bernoulli relevance models for image and video annotation. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004, vol 2. IEEE, pp II–II

  19. Feng Y, Lapata M (2008) Automatic image annotation using auxiliary text information. In: ACL, vol 8, pp 272–280

  20. Finkel JR, Grenager T, Manning C (2005) Incorporating non-local information into information extraction systems by gibbs sampling. In: Proceedings of the 43rd annual meeting on association for computational linguistics. Association for Computational Linguistics, pp 363–370. http://nlp.stanford.edu/~manning/papers/gibbscrf3.pdf

  21. Fisichella M, Stewart A, Denecke K, Nejdl W (2010) Unsupervised public health event detection for epidemic intelligence. In: CIKM ’10, pp 1881–1884

  22. FMS Inc. (2017) Sentinel visualizer. www.fmsasg.com. Accessed 30 Sept 2018

  23. GeoDataSource (2018) World cities database. www.geodatasource.com/world-cities-database. Accessed 30 Sept 2018

  24. Guillaumin M, Mensink T, Verbeek J, Schmid C (2012) Face recognition from caption-based supervision. Int J Comput Vis 96(1):64

    Article  MathSciNet  MATH  Google Scholar 

  25. Gung J, Kalita J (2012) Summarization of historical articles using temporal event clustering. In: HLT-NAACL ’12, pp 631–635. http://dl.acm.org/citation.cfm?id=2382029.2382134

  26. Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4295–4304

  27. Heath K, Gelfand N, Ovsjanikov M, Aanjaneya M, Guibas LJ (2010) Image webs: computing and exploiting connectivity in image collections. In: CVPR ’10, pp 3432–3439

  28. Heuer R (1999) Psychology of intelligence analysis. In: CIA ’99

  29. Hossain MS, Andrews C, Ramakrishnan N, North C (2011) Helping intelligence analysts make connections. In: AAAI ’11 workshop on scalable integration of analytics and visualization

  30. Hossain MS, Butler P, Boedihardjo AP, Ramakrishnan N (2012) Storytelling in entity networks to support intelligence analysts. In: KDD ’12. https://doi.org/10.1145/2339530.2339742

  31. Hossain MS, Gresock J, Edmonds Y, Helm R, Potts M, Ramakrishnan N (2012) Connecting the dots between pubmed abstracts. PLoS ONE 7(1):e29509

    Article  Google Scholar 

  32. Huang C, Ai H, Li Y, Lao S (2007) High-performance rotation invariant multiview face detection. TPAMI 29(4):671–686

    Article  Google Scholar 

  33. Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report 07–49, University of Massachusetts, Amherst

  34. IBM Analytics for a Safer Planet (2016) www.ibm.com/analytics/us/en/safer-planet/. Accessed 30 Sept 2018

  35. IN-SPIRE Visual Document Analysis (2014) http://in-spire.pnnl.gov/. Accessed 30 Sept 2018

  36. Kader MA, Boedihardjo AP, Naim SM, Hossain MS (2016) Contextual embedding for distributed representations of entities in a text corpus. In: KDD BigMine 2016, vol 53 of proceedings of machine learning research, PMLR, San Francisco, CA, USA, pp 35–50. http://proceedings.mlr.press/v53/kader16.html

  37. Kader MA, Naim SM, Boedihardjo AP, Hossain MS (2016) Connecting the dots using contextual information hidden in text and images. In: AAAI

  38. Kalva P, Enembreck F, Koerich A (2007) Web image classification based on the fusion of image and text classifiers. In: ICDAR, vol 1, pp 561–568

  39. Kang H, Plaisant C, Lee B, Bederson BB (2007) NetLens: iterative exploration of content-actor network data. Inf Vis 6(1):18–31

    Article  Google Scholar 

  40. Karpathy A, Fei-Fei L (2015) Deep visual-semantic alignments for generating image descriptions. In: CVPR, pp 3128–3137

  41. Karpathy A, Joulin A, Li FFF (2014) Deep fragment embeddings for bidirectional image sentence mapping. In: Advances in neural information processing systems, pp 1889–1897

  42. Kinsella S, Murdock V, O’Hare N (2011) “I’m eating a sandwich in glasgow”: modeling locations with tweets. In: SMUC, pp 61–68

  43. Koch GG, Koldehofe B, Rothermel K (2010) Cordies: expressive event correlation in distributed systems. In: DEBS ’10, pp 26–37

  44. Kumar D, Ramakrishnan N, Helm RF, Potts M (2006) Algorithms for storytelling. In: KDD ’06

  45. Lai JH, Yuen PC, Feng GC (2001) Face recognition using holistic fourier invariant features. Pattern Recognit 34(1):95–109

    Article  MATH  Google Scholar 

  46. Laxman S, Sastry P, Unnikrishnan K (2005) Discovering frequent episodes and learning hidden Markov models: a formal connection. KDE 17(11):1505–1517

    Google Scholar 

  47. Le D-D, Satoh S (2008) Unsupervised face annotation by mining the web. In: Eighth IEEE international conference on data mining, 2008. ICDM’08. IEEE, pp 383–392

  48. Lee H-J, Lee W-S, Chung J-H (2001) Face recognition using fisherface algorithm and elastic graph matching. In: ICIP, pp 998–1001

  49. Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5325–5334

  50. Li X, Cong G, Li X-L, Pham T-AN, Krishnaswamy S (2015) Rank-geofm: a ranking based geographical factorization method for point of interest recommendation. In: SIGIR ’15, pp 433–442. https://doi.org/10.1145/2766462.2767722

  51. Little S, Jargalsaikhan I, Clawson K, Nieto M, Li H, Direkoglu C, O’Connor NE, Smeaton AF, Scotney B, Wang H, Liu J (2013) An information retrieval approach to identifying infrequent events in surveillance video. In: ICMR ’13, pp 223–230. https://doi.org/10.1145/2461466.2461503

  52. Mishra A, Mishra N, Agrawal A (2010) Context-aware restricted geographical domain question answering system. In: CICN ’10, pp 548–553

  53. National Research Council (2002) Making the nation safer: the role of science and technology in countering terrorism. National Academies Press, Washington

    Google Scholar 

  54. OpenNLP (2017) OpenNLP. http://opennlp.apache.org. Accessed 30 Sept 2018

  55. Palantir Gotham (2007) www.palantir.com/palantir-gotham/. Accessed 30 Sept 2018

  56. Park M-H, Hong J-H, Cho S-B (2007) Location-based recommendation system using Bayesian user’s preference model in mobile devices. In: UIC, vol 4611, pp 1130–1139. https://doi.org/10.1007/978-3-540-73549-6_110

  57. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. BMVC 1(3):6

    Google Scholar 

  58. Petz G, Karpowicz M, Fürschuß H, Auinger A, Stříteskỳ V, Holzinger A (2014) Computational approaches for mining users opinions on the web 2.0. Inf Process Manag 50(6):899–908

    Article  Google Scholar 

  59. Rahman S, Naim SM, Al Farooq A, Islam MM (2010) Performance of mpeg-7 edge histogram descriptor in face recognition using principal component analysis. In: ICCIT, pp 476–481

  60. Rahman S, Naim SM, Al Farooq A, Islam MM (2012) Combination of gabor and curvelet texture features for face recognition using principal component analysis. IACSIT 4(3):264

    Google Scholar 

  61. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  62. Shahaf D, Guestrin C (2010) Connecting the dots between news articles. In: KDD ’10, pp 623–632

  63. Silva MJ, Martins B, Chaves M, Afonso AP, Cardoso N (2006) Adding geographic scopes to web resources. CEUS 30(4):378–399

    Google Scholar 

  64. Socher R, Fei-Fei L (2010) Connecting modalities: semi-supervised segmentation and annotation of images using unaligned text corpora. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 966–973

  65. Son J-W, Kim A-Y, Park S-B (2013) A location-based news article recommendation with explicit localized semantic analysis. In: SIGIR. https://doi.org/10.1145/2484028.2484064

  66. Stasko J, Görg C, Liu Z (2008) Jigsaw: supporting investigative analysis through interactive visualization. Inf Vis 7(2):118–132

    Article  Google Scholar 

  67. Stone Z, Zickler T, Darrell T (2008) Autotagging facebook: social network context improves photo annotation. In: CVPRW’08. IEEE computer society conference on Computer vision and pattern recognition workshops, 2008. IEEE, pp 1–8

  68. Sun Y, Chen Y, Wang X, Tang X (2014) Deep learning face representation by joint identification–verification. In: Advances in neural information processing systems, pp 1988–1996

  69. Sun Y, Wang X, Tang X (2013) Deep convolutional network cascade for facial point detection. In: CVPR ’13, pp 3476–3483

  70. Szegedy C, Toshev A, Erhan D (2013) Deep neural networks for object detection. In: NIPS

  71. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: CVPR, pp 1701–1708

  72. Tecuci G, Boicu M, Schum D, Marcu D (2010) Coping with the complexity of intelligence analysis: cognitive assistants for evidence-based reasoning. Technical report, LAC GMU

  73. Tian Y, Liu W, Xiao R, Wen F, Tang X (2007) A face annotation framework with partial clustering and interactive labeling. In: CVPR’07. IEEE, pp 1–8

  74. Turk M, Pentland A (1991) Eigenfaces for recognition. Cogn Neurosci 3(1):71–86. https://doi.org/10.1162/jocn.1991.3.1.71

    Article  Google Scholar 

  75. United States Government (2009) A tradecraft primer: structured analytic techniques for improving intelligence analysis. In: CIA CSI

  76. Valstar M, Martinez B, Binefa X, Pantic M (2010) Facial point detection using boosted regression and graph models. In: CVPR ’10, pp 2729–2736

  77. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: CVPR

  78. Wang D, Hoi SC, He Y, Zhu J, Mei T, Luo J (2014) Retrieval-based face annotation by weak label regularized local coordinate coding. IEEE Trans Pattern Anal Mach Intell 36(3):550–563

    Article  Google Scholar 

  79. Wu B, Ai H, Huang C, Lao S (2004) Fast rotation invariant multi-view face detection based on real adaboost. In: FG, pp 79–84

  80. Xu J, Lu T-C (2015) Seeing the big picture from microblogs: harnessing social signals for visual event summarization. In: IUI ’15, pp 62–66

  81. Yao BZ, Yang X, Lin L, Lee MW, Zhu S-C (2010) I2T: image parsing to text description. Proc IEEE 98(8):1485–1508

    Article  Google Scholar 

  82. Yong-hong T, Tie-jun H, Wen G (2005) Exploiting multi-context analysis in semantic image classification. JZUS-A 6(11):1268–1283. https://doi.org/10.1007/BF02841665

    Article  Google Scholar 

  83. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the U.S. Army Engineering Research and Development Center under Contract No. W9132V-15-C-0006.

Author information

Authors and Affiliations

  1. IBM Innovation Center, Austin, TX, 78758, USA

    Md Abdul Kader

  2. Radiant Solutions, Herndon, VA, 20171, USA

    Arnold Priguna Boedihardjo

  3. The University of Texas at El Paso, El Paso, TX, 79968, USA

    Mahmud Shahriar Hossain

Authors
  1. Md Abdul Kader
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arnold Priguna Boedihardjo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahmud Shahriar Hossain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Md Abdul Kader.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kader, M.A., Boedihardjo, A.P. & Hossain, M.S. F2ConText: how to extract holistic contexts of persons of interest for enhancing exploratory analysis. Knowl Inf Syst 61, 363–396 (2019). https://doi.org/10.1007/s10115-018-1304-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1304-9

Keywords

Search

Navigation