An effective report will address each of the research issues in turn, discussing the factors related to that issue, and introducing the research data as evidence on the issue.
A less effective (but more common) way to present reports is to give the results from each survey question, in the order that the questions were asked.
Sometimes, the two approaches produce very similar reports. The main difference is that issue-by-issue reports approach each issue from the point of view of a manager who must make a decision about something. The question-by-question reports are more like a catalogue of miscellaneous findings, often not relating each question to the actions that can be taken from its results.
A common layout for a written report is:
|
Part 1: Introduction
|
|
Part 2: Detailed findings ...for each main internal question:
|
|
Part 3: Appendix
|
Each of the three parts is intended for a different audience.
Many market research reports include an enormous set of tables in the appendix: often hundreds of pages. I don’t agree with this: if the tables are important, they should be discussed with the main part of the report. If numbers aren’t important, there’s no need to include them in the report, because nobody will read them. If they are presented without explanation, probably no readers will understand them fully. Maybe it’s best to print off a few copies of the tables, show them only to people who ask, and keep each copy in a safe, separate place — because next time a survey is done, the tables could be very useful.
For each issue: state what needs to be decided — the internal question.
List the survey questions which were chosen to give evidence on the internal question.
For each survey question:
#EG literacy by age/sex - Annotated examples
Written reports usually have three main elements: words, numbers, and graphs. A lot of research reports emphasize one of these, and pay little attention to the other two. Many research reports are mostly tables of numbers, with only a few pages of written explanation and no graphs.
I’ve found that readers of audience research reports usually have a clear preference for one type of presentation - based on their training and their work: journalists usually prefer words, accountants prefer numbers, and TV producers prefer graphs. Managers vary, depending on their training and background.
A successful report should balance all three of these components, giving similar information in various ways — but without exact repetition. This makes it easier for readers to understand.
A lot of research reports are written in this style:
74.3% of respondents agreed that the Shurple Daily had too many crossword puzzles, while 59.6% agreed that it had insufficient coverage of local news. 69.4% of women and 49% of men said that the Shurple Daily had too little local news, as did 43.1% of those aged 15 to 24, 57.8% of those in the 25-44 age group, and 65.5% of those aged 45 and over.
This is precise writing, but also difficult to understand. The reader must go over it several times, to work out exactly what it means. Bearing in mind the imprecision of surveys, there’s no point in reporting results to the nearest 0.1% - the nearest 1% is usually enough.
Here’s a simplified version of the above passage, laid out for better readability.
Readers of the Shurple Daily were asked "What do you think the Shurple Daily has too much of, or too little of?"
Too much of...
- Crossword puzzles 74%
Too little of:
- Local news 60%
What sort of people thought the Shurple Daily had too little local news?
- 69% of women
- 49% of men
- 43% of people aged 15 to 24
- 58% of people aged 25 to 44
- 66% of people aged 45 and over.
If a report is in writing, the above format makes it clearer; but if a report is spoken (e.g. as a radio program or a talk) listeners don’t get a chance to hear it again. In this case, it’s even more important to make sure the result is easily understood on first hearing. Here’s an example of wording which is difficult to understand when heard:
26% of people said they were happy with FM99’s service, while 34% said they were reasonably happy, and 31% said they weren’t happy at all. The other 6% couldn’t decide.
This is almost incomprehensible, on first hearing. Presenting the same information in a less precise (and slightly more repetitive) way actually makes it easier to understand:
When asked how happy they were with FM99’s service, listeners were divided into three groups, with about equal numbers in each. One third were very happy with FM99’s service, one third were reasonably happy, and the other third weren’t happy at all.
Bearing in mind the sampling error on most surveys, describing 26% as "one third" and not mentioning the 6% who couldn’t decide is not misleading.
21. Explaining the results in numbers
Here are some principles for presenting numbers in research reports. All of these help to communicate the findings, and make it easier for readers to understand the data correctly.
There are many different types of graph or chart, but most are not used in audience research. Those used most often include:
Some other types of charts, well suited to audience research, but less often used, include
Though many different kinds of graph are possible, if a report includes too many types, it’s often confusing for readers, who must work out how to interpret each new type of graph, and why it is different from an earlier one. I recommend using as few types of graph as are necessary.
If you have a spreadsheet or graphics program, such as Excel or Deltagraph, it’s very easy to produce graphs. You simply enter the numbers and labels in a table, click a symbol to show which type of graph you want, and it appears before your eyes. These graphs are usually not very clear when first produced, but the software has many options for changing headings, scales, and graph layout. You can waste a lot of time perfecting these graphs. Excel (actually, Microsoft Graph, which Excel uses) has dozens of options, and it takes a lot of clicking of the right-hand mouse button to discover them all. If you don’t have a recent and powerful computer, Excel can be a very slow and frustrating program to use.
The main types of graph include pie charts, bar charts (histograms), line charts, area charts, and several others.
A round graph, cut (like a pie) into slices of varying size, all adding to 100%. Because a pie chart is round, it’s useful for communicating data which takes a "round" form: for example, the answers to "How many minutes in each hour would you like FM99 to spend on each of the following types of program...?" In this case, the pie corresponds to a clock face, and the slices can be interpreted as fractions of an hour.
Pie charts are easily understood when the slices are similar in size, but if several slices are less than 5%, or lots of different colours are used, it can be quite difficult to read a pie chart. In that case the chart has to be very big, taking perhaps half a page to convey one set of numbers. Not a very efficient way to display information.
Also known as a histogram. A very common type of graph, easily understood. But when one of these charts has more than about 6 vertical bars, there’s very little space below each bar to explain what it’s measuring.
Exactly like a vertical bar chart, but turned sideways. The big advantage of the horizontal bar chart is that you can easily read a description with more than one word. Unfortunately, most graphics software displays the bars upside down — you’re expected to read from the bottom, upwards to the top. A standard bar chart looks like this. (Like the two above charts, this was created with Excel.)
Luckily, you don’t need graphics software to produce a horizontal bar chart: you can do it easily with a word processing program. One of the easiest ways to do this is to use the | symbol to produce the bars. This symbol is usually found on the \ key; it is not a lower-case L or upper-case I or number 1. It stands out best in bold type. This is what I call a blobbogram. For example:
| Q14. SEX OF RESPONDENT | ||
| Male | 47.4% | ||||||||||||||||||||||||| |
| Female | 52.6% | ||||||||||||||||||||||||||| |
| Total | 100.0% | = 325 cases |
If each symbol represents 2% of the sample, you can usually fit the graph on a single line. Round each figure to the nearest 2% to work out how many times to press the symbol key. In the above example, 47.4% is closer to 48% than to 46%, so I pressed the | key 24 times to graph the percentage of men. This is a very clear layout, and quick to produce, so it is well suited to a preliminary report.
A more elaborate looking graph can be made by using special symbols. For example, if you have the font Zapf Dingbats or Wingdings, you can use the shaded-box symbol: q
This is wider than the | symbol, and no more than about 20 will fit on a normal-width line, if half the line is taken up with the description and the percentage. Therefore, one q should be equivalent to 5%:
| Q14. SEX OF RESPONDENT | ||
| Male | 47.4% | qqqqqqqqq |
| Female | 52.6% | qqqqqqqqqqq |
| Total | 100.0% | = 325 cases |
Like a bar chart, a pictogram can be either vertical or horizontal, but instead of showing a solid bar, a pictogram shows a number of symbols - e.g. small diagrams of people. In fact, the above bar chart with the q symbol is a crude type of pictogram. But unlike a bar chart made of entire q symbols, pictograms show partial symbols. If one little man means 10%, and the number to be graphed is 45%, you see four and a half little men...
You won’t find this mentioned in books on statistics, because I made it up. It was an invention that seemed to be required, and is best described as a two-dimensional pictogram. It’s named after the game of dominos, in which each piece has a number of round blobs in a rectangular block.
Just as you use a bar chart or pictogram when graphing one nominal variable, a domino chart is used to compare two nominal variables — it’s the graphical equivalent of a cross-tabulation. It is used to show the results of inquiries such as "Do more men than women listen to FM99?" In this case, the two variables are sex and listening to FM99.
Suppose 71% of respondents listen to FM99, and 52% are men. To find the sex breakdown of FM99 listeners, you need to produce a crosstab. The resulting table might look like this:
|
Sex |
Listen to FM99? |
Total |
|
|
Yes |
No |
||
|
Male |
40% |
12% |
52% |
|
Female |
31% |
17% |
48% |
|
Total |
71% |
29% |
100% |
To produce the domino chart, take each percentage in the main part of the table (not counting the Total row or column), divide each figure by 4, round it to the nearest whole number, and type in that number of blobs. "Why divide by 4?" you may wonder. It’s because each blob is equivalent to 4% of the people answering both questions. The figure doesn’t need to be 4; it could be 5 or 2, though a 5% blob often isn’t quite detailed enough, while 2% produces so many blobs that it’s harder to interpret the table at a glance. Because one blob equals 4%, there should be 25 blobs in the whole table — though occasionally, due to rounding, there will be 24 or 26.
Here’s a domino chart of the above table:
|
Sex |
Listen to FM99? |
|
|
Yes |
No |
|
|
Male |
••••••••• |
••• |
|
Female |
•••••••• |
•••• |
Though the chart has less detail than the table, most people can understand it instantly. It shows that slightly more men than women listen to FM99.
Domino charts are specially useful when you have a group of tables. and you want to compare the answers. A lot of domino charts can fit onto a single page. The readers’ eyes will be drawn to the cells with the largest and smallest numbers of blobs.
This is a very simple graph: easily understood, and easily produced. Though graphics software doesn’t do domino charts (yet) you can create a domino graph with the blob symbol in a word-processing program.
This is used when the variable you are graphing is a numeric one. In audience research, most variables are nominal, not numeric, so line charts aren’t needed much. But to plot the answers to a question such as "How many people live in your household?" you could produce a graph like this:
It’s normal to show the measurement (e.g. percentage) upwards, and the scale (e.g. hours per week) on the horizontal scale. Unlike a bar chart, it will confuse people if the scales are exchanged. You’ll find that almost every line chart has a peak in the middle, and falls off to each side, reflecting what’s known as the "normal curve."
A line chart is really another form of a vertical bar chart. You could turn a vertical bar chart into a line chart by drawing a line connecting the top of each bar, then deleting the bars.
A line chart can have more than one line. For example, you could have a line chart comparing the number of hours per week that men and women watch TV. There’d be two lines, one for each sex. Each line needs to be shown with a different style, or a different colour. With more than 3 or 4 lines, a line chart becomes very confusing, specially when the lines cross each other.
In a line chart with several lines — such as the above example, with two sexes — each line starts from the bottom of the table. That way, you can compare the height of the lines at any point. An area chart is a little different, in that each line starts from the line below it. So you don’t compare the height of the lines, but the areas between them. These areas always add to 100% high. You can think of an area chart as a lot of pie charts, flattened out and laid end-to-end.
A common use of area charts in audience research is to show how people’s behaviour changes across the 24 hours of the day. The horizontal scale runs from midnight to midnight, and the vertical scale from 0 to 100%. This area chart, taken from a survey in Vietnam, shows how people divide their day into sleep, work, watching TV, listening to radio, and work and everything else.
An area chart needs to be studied closely: the results aren’t obvious at a glance. However, area charts provide a lot of information in a small space.
There are dozens of other chart types not mentioned above, and also dozens of variations on the above types - specially bar charts. However the above graph types cover most situations. It becomes confusing to readers of reports if many different types of graph are presented, so I recommend that any report should include no more different graph types than necessary.
The most appropriate type of graph to present depends on the number of variables being displayed, and whether these are nominal variables (with a limited number of separate values) or metric variables (whose value can be any number). My suggestion is to use a horizontal bar chart whenever possible. In a normal audience survey, less than a third of the graphs are unsuited to being shown as horizontal bar charts.
|
Variables |
Recommended chart type |
|
|
number |
type |
|
|
1 |
nominal |
bar chart, pictogram, or pie chart |
|
1 |
metric |
line graph, or box and whisker plot |
|
2 |
both nominal |
multiple bar chart, or domino chart |
|
2 |
both metric |
bubble chart, or scattergram |
|
2 |
1 metric, |
box and whisker plot, or area chart |
3-D charts can look very impressive, but I strongly suggest you avoid using them — it’s just too easy to misread them. The simpler a graph is, the more effective it is at communicating
Enough information needs to be supplied to enable informed readers to judge the likely accuracy of the results. Therefore, you need to include this type of information about how the research was done:
The above list is a long one, and the background data about a research project can fill many pages if you include a lot of detail. However it’s possible to include most of this information in one paragraph — for example:
"This information comes from a telephone survey, with interviews from 1050 people aged 18 and over living in the Adelaide region. The research was done between 5 and 19 May 2000 by Audience Dialogue, using interviewers with at least 40 hours' training. The response rate was 77%. The survey was commissioned by radio FM99, but respondents were not told this. It’s possible that the results were influenced by FM101, which reported on 17 May that the survey was taking place, and urged respondents to say that they only listened to FM101. Because only a sample of the population was included, figures in this survey are likely to vary from the true population figures by about 1.5% on average."
That explanation provides the information that well-informed readers need, to judge the likely accuracy of the survey.
One area of argument is whether a research report should include recommendations. The survey analyst, after spending weeks with the data, understands it better than anybody else ever will, and is in a good position to recommend certain courses of action.
Usually, though, the analyst is not aware of all the constraints on action, so these recommendations are seen as impractical and often are not acted on. On the other hand, managers will often dismiss recommendations as impractical, simply because they have not considered them in detail.
Some researchers believe their job is only to produce the results, and that it is up to users to make any recommendations. My experience is that inexperienced buyers of research find it difficult to draw recommendations from research data, and if the researcher doesn’t make recommendations, the results of the research are often not acted on.
I’ve found that recommendations are best made not by the researcher or users alone, but by both groups working together. Soon after a survey report has been sent out, arrange a workshop session in which the implications of the survey can be discussed, and recommendations formed, or decisions made. This sort of workshop can last from a few hours to a whole day - depending on the number of questions in the survey, and how much disagreement is expressed.