As James and I were chatting recently, it became clear that whilst we both call ourselves analysts, there are some important differences in what we do and how we spend our time. James therefore asked me to put a few thoughts together on the role of a quantitative research analyst, (or “numbers analyst” as he calls it) and how to become one.
So, to begin with, what do us numbers guys do?
At a simplistic level, we design and execute surveys, then analyse and interpret the results, which may then be used as input into business planning and development activity or as the basis for market analysis reports and advisory services. It sounds relatively straightforward and at a high level it is, but the devil is in the detail, so it’s worth talking through the process to get a feel for the kinds of things a numbers analyst needs to deal with.
Given a particular requirement or brief, for example, the first challenge is designing the study. Considerations here are sampling of target respondents, qualification of respondents for inclusion in the study, positioning of the interview, and method of execution (face to face versus telephone versus online).
All of these have an impact on the result. To use a an extreme case to illustrate the point, if the objective is to investigate a particular aspect of Linux and you use an online survey enticing people to participate by saying “Give us your views on Linux in the context of …”, Linux advocates will be much more likely to respond than sceptics or agnostics. This gives you a skewed sample which clearly means it is invalid to subsequently say things like “X% of companies think Linux is great for ….”. However, in a study of this nature, if you have asked the right questions, you would be able to make statements confidently and legitimately about those with experience of relevant use or differences in perceptions between users and non-users.
And this brings us on to the important area of questionnaire design. This is a massive topic and anyone on the receiving end of a poorly designed questionnaire executed by a pure market research company will understand this. In IT related research, questionnaires really need to be designed by industry analysts with a knowledge of the area being investigated, a feel for how the data will ultimately be analysed, and experience of “choreographing” the interview flow in a way that builds rapport and elicits unambiguous answers.
Execution is a topic I won’t go into here (i.e. the project management and operational side of actually gathering the data), but suffice it to say that this is pretty involved and unless it is handled well, the end result is likely to be compromised.
Finally, a numbers analyst will work with the data, analysing the obvious, but also looking for those trends and correlations that provide the real insights into what’s going on. Again, this can only be conducted effectively by an industry analyst with the necessary industry knowledge, otherwise important findings are missed and/or observations are misinterpreted – a common occurrence, for example, when PR companies engage generic market research firms and come out with ludicrous headlines for press releases.
Jaime Cardoso says:
January 27, 2006 at 8:06 pm
Having had a lot of coleagues that went into the Statistics field of work, I can only say that I totally understand Dale’s comment.
I remember when a company ordered a study of customer satisfaction and, on the other hand, it gave the Market Research Firm the desired results. They had a lot of work comming up with the right questions to get the desired results but, all the interviews were conducted and the data was processed normally.
ARonaut says:
January 30, 2006 at 4:43 pm
James, You seem to pigeonhole numbers analysts into those doing surveys, forgetting IDC, Gartner (Dataquest), Canalys, Context, etc… They produce market share information and tend to be called “numbers analysts” more than those just doing surveys…
James Governor says:
January 31, 2006 at 8:19 am
thanks ARonaut, that’s a good point. pigeonhole is a good work considering you’re talking about the “bucket game”
bear in mind that IDC, at least, also carries out ROI/TCO studies – which is probably another addition to the numbers analyst taxonomy. the post was never intended to be once and future, so feedback is welcome.
taxonomy
1. market research
2. market share analysis
3. ROI and TCO studies
your comment does indicate a bias on your part however, talking of pigeonholing. more than those “just doing surveys”…?
Where do you think IDC and Gartner get their numbers though? they do surveys of major vendors. i dont know about canalys.
ARonaut says:
January 31, 2006 at 5:37 pm
James, IDC and Dataquest (the two 800 pounds gorillas) get their numbers from polling vendors and surveying both the channel and users (IT budget surveyes).
The difference between polling and surveying is the sample size -polling usually includes all market participants. This has a direct effect on statistical accuracy (dispersion and error margins): on some commodity markets (mobile phones, pc’s, etc), IDC and Gartner/Dataquest frequently come within 1/2 point of each other. On application software and other “licenced goods”, market numbers are usually much harder to track.
Why don’t you invite IDC or Gartner to do a guest post?
James Governor says:
February 1, 2006 at 6:16 pm
i tend to think IDC and Gartner’s can do their own comms. why don’t they offer, or post themselves? if someone wants to offer i won’t cut it. but i am not going to go out of my way in that regard.
I am also still skeptical about what you’re saying. poll size? The market share numbers in many cases come from polling the vendors, not the market. that is a SMALL sample set, not a large one. If you are claiming they do a correlation between market polling and market share ok – but i have my doubts, and many vendors i have spoken to do as well.
if they really poll the market, why are the numbers on OSS technology so whack?
what you said – they can track boxes based on vendor claims – thus the agreement between the two firms. when it comes to real market measurement i am far more skeptical.
anyone from IDC or Gartner can comment here- why don’t *you* ask them to post something.
Arnie McKinnis says:
February 1, 2006 at 10:18 pm
From Data to Wisdom….
We collect data
Add context to get information
Add understanding to get knowledge
Add judgement and values to get wisdom.
Interesting Post. IMHO, most industry “experts” never really get past the information stage – to creating actual knowledge, let alone going all the way to wisdom. Right, wrong or indifferent – this happens regardless of the approach
James says:
February 4, 2006 at 2:39 pm
Is it possible to be quantitative when covering open source software or does this only apply to proprietary vendors?
When will RedMonk merge with others so that corporate buyers can get all their information from a single source?
James Governor says:
February 7, 2006 at 5:52 pm
no intention to merge with anyone. wouldnt need to to provide a single library, anyway.
i will be putting forward some models to solve the OSS market share issue soonest… wait and see.