SAMT08
Semantic-driven multimedia retrieval with the MPEG Query Format
Ruben Tous and Jaime Delgado
Distributed Multimedia Applications Group (DMAG) Universitat Politcnica de Catalunya (UPC)
Dept. Computer Architecture Barcelona (Spain)
1
Presentation outline
The MPEG Query Format (MPQF) MPQF evaluation model How to extend MPQF to allow SPARQL-like queries MPQF and metadata interoperability
MPEG Query Format (MPQF)
MPEG Query Format (MPQF): Introduction
ISO/IEC 15938-12 Query Format (MPQF)
Recently reached the International Standard level
MPQF: Provides a standardized interface for multimedia content retrieval systems (e.g.: MPEG-7 databases) Allows combining Data Retrieval like conditions (XPath, XQuery) with Information Retrieval like conditions (Query-by-Example, etc.) Is an XML application: One XML schema describing messages:
Requester -> responder = Input Query Format Responder -> requester = Output Query Format Management messages (service properties, etc.)
Part 12 of ISO/IEC 15938 (MPEG-7), but METADATA-NEUTRAL
4
Input Query Format Output Query Format
Requester
Query Management Input Query Management Output
Responder
MPQF
Query Splitter
Service 1
Client
MPQF
Service Provider
Resultlist Aggregator
MPQF
Service 2
MPQF
Service N
MPEG Query Format (MPQF)
Schema overview of the uppermost elements of MPQF
Input Query FetchResult Output MpegQuery Input Management Output
MPEG Query Format (MPQF)
Input Query Format (IQF)
QFDeclaration OutputDescription Input QueryCondition ServiceSelection
Path TargetMediaType Join Condition
MPEG Query Format (MPQF)
DR-like criteria Boolean connectors XML query algebra (metadata-neutral) DR-like criteria Embedded XQuery expressions (metadata-neutral)
Condition types
QueryByFreeText IR-like criteria QueryByDescription QueryByMedia SpatialQuery TemporalQuery QueryByROI
8
MPEG Query Format (MPQF)
Example input query condition tree (graphical view)
<InlineMedia type=image/jpeg"> <MediaData64>R0lGODlhDwAPAKECAAAAzMzM///// wAAACwAAAAADwAPAAACIISPeQHsrZ5ModrLlN 48CXF8m2iQ3YmmKqVlRtW4MLwWACH+H09 . </MediaData64> </InlineMedia>
//FileSize
1000
QueryByMedia
<
AND
Input Query Format (Example)
10
MPEG Query Format (MPQF)
Output Query Format (OQF)
Comment GlobalComment Output ResultItem SystemMessage TextResult Thumbnail MediaResource Description AggregationResult
11
Output Query Format (Example)
12
MPQF evaluation model
Multimedia Database: Simplistic view
image1
<metadata>
image2
<metadata>
image3
<metadata>
14
Multimedia Database: Complex view (the one in the Query Format)
Content DB
Metadata DB
image1
<metadata>
image1
<metadata>
image2
<metadata>
i3.1
i3.2
<metadata>
image4
<metadata>
15
The Query Format assumes XML metadata
XML
<metadata>
16
MPQF in depth
The EvaluationPath element
<MpegQuery> <Query> <Input> <QueryCondition> <EvaluationPath>//VideoSegment</EvaluationPath> <Condition xsi:type="QueryByFreeText"> <FreeText>Lausanne</FreeText> </Condition> </QueryCondition> </Input> </Query> </MpegQuery>
17
Multimedia Database: Complex view (the one in the Query Format)
Content DB
Metadata DB
Image/regionlocators//IDs
image1
image1
image2
i3.1
i3.2
18
Image Database: Practical view (ONE BIG XML TREE)
Content DB
Metadata DB
Independent metadata
image1
image2
Imageregion locators
i.1
i3.2
MPQF in depth
Two evaluation styles in MPQF: Boolean and fuzzy-logic
input sequence
//FileSize < 1000
QueryByMedia
1=TRUE 1=TRUE 1=TRUE output sequence 0.56
0.4 0.7
20
How to extend MPQF to allow SPARQL-like queries
How to extend MPQF to allow SPARQL-like queries
Knowledge base (KB) = Tbox (terminological component) +Abox (assertion component) Some relevant initiatives are choosing the RDF language for modeling metadata metadata Typical Semantic Web based multimedia KB (e.g. PhotoRDF): Tbox = RDFS/OWL ontologies Abox = RDF statements about the media objects
ABox s s s p p p p p p p p o o o o o o o o TBox(ontologies)
Knowledge DB
s s s s s
22
How to extend MPQF to allow SPARQL-like queries
SPARQL. W3C Recommendation on 15th January 2008 An SPARQL query consists of a basic graph pattern, expressed as a list of triple patterns
PREFIX dc: <http://purl.org/dc/elements/1.1/> PREFIX ns: <http://example.org/ns#> SELECT ?title ?price WHERE { ?x ns:price ?price . p o s FILTER (?price < 30.5) p o s ?x dc:title ?title . } p o s
s s p p o o
23
How to extend MPQF to allow SPARQL-like queries
Proposal 1/3: Extend the MPQF data model
Content DB Metadata DB Knowledge DB
ABox TBox(ontologies) o o o o o o o o
video1
s s s
p p p p p p p p
video2
s s s s s
v3.1
v3.2
24
How to extend MPQF to allow SPARQL-like queries
Proposal 2/3: Include a new query type: QueryBySPARQL
<MpegQuery mpqfID="someID"> <Query> <Input> <QueryCondition> <Condition xsi:type="QueryBySPARQL"> <SPARQL> <![CDATA[ PREFIX dc: <http:///> ASK { ?resource dc:title "Barcelona . } ]]> </SPARQL> </Condition> </QueryCondition> </Input> </Query> </MpegQuery>
25
How to extend MPQF to allow SPARQL-like queries
Visual representation of the execution of the new QueryBySPARQL query type
Content DB Metadata DB Knowledge DB
EvaluationItem(EI) video1
s s s p p p p p v3.1 v3.2 s p s p p o o o o o o
video2
matchingRDFtriples
26
<InlineMedia type=image/jpeg"> <MediaData64>R0lGODlhDwAPAKECAAAAzMzM///// wAAACwAAAAADwAPAAACIISPeQHsrZ5ModrLlN 48CXF8m2iQ3YmmKqVlRtW4MLwWACH+H09 . </MediaData64> </InlineMedia> PREFIX dc: <http://purl.org/dc/elements/1.1/> ASK { ?resource dc:title "Barcelona . }
QueryBySPARQL QueryByMedia
//FileSize
1000
<
AND
But, what about metadata interoperability with MPQF?
MPQF and metadata interoperability
Currently MPQF is metadata-neutral Queries refer to paths related to different metadata formats Several metadata formats exist: Dublin Core, EXIF, ID3, MPEG-7, etc. Semantic mappings can help circumventing the problem The mappings are (supposed to be) easiest to specify with ontologies (with our proposal) we can use RDF and SPARQL in MPQF but Can I use ontologies with MPQF? (currently) no, at least not in a standard way
29
MPQF and metadata interoperability
Proposal 3/3: Extend MPQF: Allow registering RDFS/OWL ontologies Be aware of the registered ontologies when querying XML Be aware of the registered ontologies when querying RDF Be aware of the registered ontologies when querying free text?
30
MPQF and metadata interoperability
Two choices to achieve interoperability with N formats: Specify NxN mappings Specify one reference metadata format and N mappings (better!) Which kind of reference metadata format? An XML Schema (e.g. XMP) But mappings are simpler if the reference metadata model is also an ontology This issue is now being discussed at JPEG (JPSearch initiative) OPEN ISSUE
31
Conclusions
The MPEG Query Format is finished! But now is time for amendments and corrigenda The implementation race has begun MPQF lacks the ability of handling RDF and SPARQL Proposal 1/3: Extend the logical model Proposal 2/3: New query type: QueryBySPARQL Proposal 3/3: Allow registering ontologies A relevant piece is still missing: A pivot metadata format
32
Acknowledgments
Thank you
33
