[Top] [All Lists]

Re: [ontolog-forum] Ontologist Aptitude Test?

To: "[ontolog-forum]" <ontolog-forum@xxxxxxxxxxxxxxxx>
From: Joel Bender <jjb5@xxxxxxxxxxx>
Date: Fri, 18 Dec 2009 17:43:11 -0500
Message-id: <00D02C81-372A-4E92-9751-C8A72CC5ED02@xxxxxxxxxxx>
John,    (01)

> That is most definitely false.    (02)

Wow!  Whenever I've asked somebody that said they've improved something, 
they've always given me some measure.  If they are about to work on a problem, 
I've always asked them "how do you know when you've made any progress?"  
Sometimes they hadn't though much about how to measure how effective their work 
will be.    (03)

How do *you* measure success?    (04)

> As for ontology, this forum shows that the field itself is extremely 
>unsettled, and there is no consensus on what a good ontology is or how it 
>could be measured.    (05)

I'll grant you that ontology as a field is unsettled, but it's not without 
descriptive statements that say what it is.  Are there no questions that can be 
asked that test a persons knowledge of those statements?    (06)

Any specific ontology can only be measured as "good" against a specific use 
case, and if there is something that is better, there must be a reason it is 
better.    (07)

> I was thinking of a team of people that I would hire to solve a problem.  And 
>passing a multiple choice test gives ZERO evidence of problem solving ability.    (08)

A wild overgeneralization which tells me you have no confidence in the way 
tests are built.  I had a physics test that was all multiple choice; the 
answers were similar to (a) 73.5, (b) 73.6, (c) 73.7, (d) 89.7, (e) None of the 
above.  The chances of 1 in 5 of getting the right answer (and there were a lot 
of questions) was enough for the professor and saved enormous time reviewing 
answers!  The answers to essay questions are easily filled with puffery, which 
is subject to vast chasms of interpretation, and why engineers have such a hard 
time dealing with English majors.  :-)    (09)

Besides, I wasn't specifically limiting my quest for possible questions to only 
multiple choice ones.    (010)

> I certainly would not hire somebody whose primary credential was the ability 
>to pass a multiple-choice test.    (011)

True, and I wouldn't hire anybody without at least a baccalaureate degree, and 
I would expect their degree program to include other kinds of questions as well.    (012)

> The best criterion is successful performance on real jobs.  Second best is an 
>interview with somebody in the field.  A test is almost worthless.    (013)

Please don't say that to your pharmacist.  Later you write:    (014)

> Doug Lenat did use a simple test, but only to weed out the incompetent.    (015)

Weeding out the incompetent is a useful process, it allows 90% of the 
applicants to be flushed.  Anything that saves me time and money in finding 
somebody that can solve my problem is a good thing.    (016)

> Not the MCP exams that Microsoft uses.  There are many very competent people 
>who have an MCP certificate.  But that is because they took the exam in self 
>defense to avoid looking worse than the idiots whose only knowledge is the 
>content of the MCP books.    (017)

Like it or not, the industry has developed these tests specifically for that 
purpose, and people that don't bother to take those tests *do* look worse than 
the idiots, even if they're not.  They are being measured.  Maybe that specific 
measuring tool isn't so great.  Maybe they *are* better than all of the other 
applicants that have taken the test, but am I really supposed to go through all 
of my applicants, hunting for that special someone?    (018)

You should take a look at the CCNE process from Cisco and their tests, they are 
far from worthless.  Please don't throw out the test and certification process 
baby with the Microsoft bath water.    (019)

> JB> ... the design and development teams saying "we need an ontologist"
>> or he/she goes to lunch with a colleague that says "we've done so much
>> better after we hired an ontologist".
> On what planet have you ever heard anybody say anything like that?    (020)

I haven't heard that, but are you implying that nobody on this planet has ever 
said anything like that?  That doesn't bode well for this particular field of 
study, and I also don't believe it.  There exists somebody somewhere that has 
played the role of ontologist and made some improvement to something, and that 
improvement was measured and documented.  That role involved skills beyond the 
systems analyst, maybe it included philosophy, biology, or epistemology.  What 
were those skills?  How do I know you've got them?    (021)

> I would ask them the same question that Lou Gerstner asked the VPs who 
>reported to him when he was named the CEO of IBM.    (022)

Another good "team building" scenario, but I don't need a VP.  I don't need an 
athlete, I need a wide receiver, and if they can't do a 100 yard dash in X 
seconds, they're out.  If they ran better than that at the last track meet, and 
I trust the time keepers, I don't have to test them for that.  They were 
measured.  If they say their times have improved, that was measured too, or I 
say they haven't.    (023)

Maybe I need a new kind of person, a lot like the systems analysts that have 
been around for 30 years, but someone with a different perspective.  (When I 
worked for a small consulting firm we had a psychiatrist on staff, not for his 
computer skills, but because he could connect with the customers in a way the 
technical staff couldn't.)    (024)

Maybe I need an ontologist.    (025)

> He gave all prospective candidates a list of six sentences in English, which 
>he asked them to translate to FOL.
> But this test was just to verify a minimal competence.  It didn't produce a 
>number that he used to rank the candidates.    (026)

Excellent!  If they couldn't answer any of the questions, they were in serious 
trouble.  If they answered them all, they were clearly on to the next round of 
testing.  If the sentences were carefully constructed by people that understood 
some of the traps that people doing English -> FOL translation can fall into, 
and the responses were independently verified as accurate and complete, that 
would be even better, I could skip that part of the interview.    (027)

If they don't want to be bothered with taking the test then I have more than 
enough applicants that are, and they're just doing me a favor because I don't 
want somebody with that attitude.    (028)

Joel    (029)

Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/  
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/  
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/ 
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx    (030)

<Prev in Thread] Current Thread [Next in Thread>