Re: [ontology-summit] OntologySummit2011: Track 3 - Value Metrics and Va

[Chris Baker <denguehost@xxxxxxxxx>]

Hello

I comment below on #3 and #5 in Todd and Rex's list of metrics

I think the key is that in a given use case there must be a benchmark of the existing systems performance. Only then can the benefit of an ontology based system be validated or shown to be cost effective or revenue yielding. 

In my use case - Technical Product Support in the Contact Centre the baseline is the time it takes for an agent to find a PDF file and doing keyword word search inside it (which is the state of the art). Our semantic solution saves an agent upwards of 50% time savings and enables Tier 1 agents to achieve more complex tasks. 

RE: Metrics - Call centres actually have a series of well established metrics that the semantic solution must illustrate impact on for it to be adopted. These include

Utilization (Productive versus non-productive time) 
First Call Resolution
Case Closed Timeframe (Total elapsed time)
Filtration Rate (Escalation to Tier 2) [Linked to First Call Resolution]

Apologies if you have seen this before: 

Testing Times for the Contact Centre and Semantics
http://videolectures.net/estc2010_baker_ttfcc/

Funded by the Atlantic Canada Opportunities Agency and the Atlantic Innovation Foundation, Innovatia Inc. have pioneered the design and testing of semantic technologies for use by Contact Centre agents who provide Technical Support to customers in the Telecommunications sector. This is in response to increasing contact center costs for companies whose products and information support services must rapidly evolve. Numerous opportunities exist for increasing the productivity of knowledge workers involved in searching separate and disconnected product-specific knowledge bases, case resolution databases, training manuals and technical documentation. Our technical solution comprises of OWL-DL knowledge base populated from a wide variety of document formats with sentence-triples generated by a telecommunications-specific text mining pipeline that leverages document segmentation techniques to instantiate with state of the art precision and recall. This talk will outline critical features of the platform and the performance of agents using the prototype with various user interface tools from commercial vendors with customized features to test this search paradigm. Pilot studies required agents to troubleshoot customer queries from fault symptoms to root causes and to identify procedures to resolve the causes of faults. During testing agents used visual queries, advanced dynamic form-based search, and SPARQL queries saved as frequently asked questions. In particular we have validated that Tier 1 agents can navigate a sequence of workflows resulting in an increase in findability from 75 to 100% under real time conditions and overall search time has been reduced by 50%. Moreover, time savings can be made at the critical junction of case handoff from Tier 1 to Tier 2 agents, such that Tier 1 agents can achieve better performance. We review contact center performance metrics and comment the on the suitability of Semantic Technologies for this business process as well as issues related to rollout and adoption of this platform across the enterprise. 


On Tue, Feb 1, 2011 at 9:32 PM, Todd J Schneider <todd.schneider@xxxxxxxxxxxx> wrote:
>
> Track 3 of the 2011 Ontology Summit is focused on Value Metrics and Value
> Models that can be used to make the case for the use of ontologies and
> semantic technologies.
>
> Toward this end we are soliciting panelists and input for Value Measuring
> Methodologies (Metrics),
> how those measurements are made and what those measurements mean.
>
> We also seek input on what constitutes a Value Model and how metrics work
> within those models.
> Among the areas where metrics would be useful are
>
> 1.      Ontology Development - What are the costs/risks associated with
> developing ontologies (NOTE: development could also include reuse of
> existing ontologies).
>
> 2.      Ontology Integration - To what extent is the use of ontologies or
> related technologies integrated into an overall development/project? To
> what extent can existing ontologies be integrated into existing Business
> and IT systems. What risks are associated with their use?
>
> 3.      Performance - How much/well does the ontology or related semantic
> technologies improve or impede performance?
>
> 4.       Deployment - To what extent does the use of ontologies or their
> attendant technologies aid or hinder deployment?
>
> 5.      Operation - To what extent does the use of ontologies or related
> semantic technologies improve or hinder Business Operations, reduce
> Production and worsen Quality of Service/Product?
>
> 6.      Maintenance – To what extent does the use of ontologies or their
> attendant technologies improve or worsen the maintenance of services,
> systems or products?
>
> Cheers,
> Rex & Todd
>
>
> _________________________________________________________________
> Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
> Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
> Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
> Community Files: http://ontolog.cim3.net/file/work/OntologySummit2011/
> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011
> Community Portal: http://ontolog.cim3.net/wiki/



--
Christopher J. O. Baker Ph. D.
Associate Professor
Dept. Computer Science and Applied Statistics
University of New Brunswick, Canada
http://ca.linkedin.com/in/christopherjobaker
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2011/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011  
Community Portal: http://ontolog.cim3.net/wiki/     (01)