How you can create a 'franchise' survey

Here's advice from Folio magazine editor Tony Silber on conducting a high-quality reader survey.

by Martha Spizziri, President, Boston chapter

A well-conducted reader survey on a hot topic can position your publication as an expert on the subject. The survey can become an eagerly awaited event, boosting readership and capturing the attention of advertisers.

At the ASBPE East Coast regional awards banquet held earlier this year, New York chapter president Lee McDonald moderated a panel on how to conduct surveys. In this issue, Folio editor Tony Silber reveals his magazine’s approach to creating successful surveys. We will publish additional articles about surveys in future issues.

How do you publish the same series of surveys every year and still keep it fresh?

That’s the challenge Folio magazine faces with its annual salary surveys of the magazine industry. It publishes four each year: a production survey in January, an editorial survey in August, ad sales in September, and circulation in December.

The Folio staff works hard to maintain the quality of their surveys. Collecting and analyzing the data for each one is a months-long process.

"Surveys create a dominant editorial franchise," says Folio editor Tony Silber. "They make [your publication] an authority, and they confer ‘must-read’ status."

But Silber points out that if the quality of the data isn’t there, the quantity is unimportant. A survey has to have "visceral believability," he says. "If you publish results that don’t bear out actual conditions in the industry, readers will notice and you’ll lose credibility."

Silber suggests subjecting your data analysis to a reality check. "Ask yourself, ‘Given what I know about the industry, does this feel right?’ You can easily lose sight of this when you’re working with all this data," he points out. If the results don’t seem consistent with reality, that could be a clue that they’re faulty — or that deeper analysis is needed to explain the anomaly.

For example, Folio’s surveys were showing salaries for the Northeast that were higher than what the editors’ knowledge of the industry would indicate. Realizing the results were being skewed upward by New York City, they asked for separate data on New York on subsequent surveys.

Similarly, tempting as it might be, you can't make statements about year-to-year trends based on a comparison of current survey data and previous years' data. You'll be comparing two entirely different groups of respondents, Silber points out. In order to make valid year-to-year comparisons, you must ask for historical data in each year’s survey so you can compare current and historical figures from the same group of respondents.

In addition, data must be weighted and cross-tabulated. In the Folio surveys, the circulations of the magazines sampled must reflect the actual distribution in the industry, or generalizations drawn from the data are likely to be flawed.

Other credibility pitfalls:

• A bad list
• Poorly phrased questions
• Errors in transcribing numerically coded answers
• Improper analysis (You need an editor to do the analysis, Silber advises.)

Seven Steps For A Successful Survey

Even before you deal with the nitty-gritty statistical details, you should step back and look at the big picture. Here are some key actions to take.

1. Decide the objectives of your survey. Obvious though it may seem, many editors overlook this step. Don’t. It will help ensure that you collect the right data to do the analysis you want to do.

For Folio, objectives included statistical validity and the ability to do year-to-year comparisons, and cross-tabulate many different data points. Often, surprising information comes from gathering all this information. For instance, the Folio editors found that having responsibility for multiple magazines doesn’t necessarily boost one’s salary, but supervising a large number of people does.

2. Decide whether the survey will be done in house or by an outside firm. "This depends largely on how ambitious you want to get with your survey," says Silber. Folio’s first survey was completely outsourced — including the analysis. The staff soon realized this was a mistake. An editor’s judgment and knowledge are crucial. Although they still use outside help for some parts of the process, the editors now analyze the data themselves.

3. Decide on an editorial direction for the survey. This includes the survey’s scope, the type of data you want to capture, and how you want to present it in the magazine.

4. Get a project manager. "You need someone who can keep the mechanical process on track," advises Silber. The project manager doesn’t have to have an editorial vision, or even know what that vision is, but it certainly helps.

5. Get a good writer and a good designer. "They have to know how to present data so it’s interesting, easy to understand, and makes sense," Silber says. But more than that, they must know what’s statistically important and what’s important to the audience, and be able to figure out what the underlying message of the statistics is.

6. Let your surveys evolve! This has proven to be a must for Folio. "We force ourselves to rethink the objectives every year," Silber says. For instance, the staff decided they wanted to track salary trends by job title. So, they started asking for five-year historical data from respondents whose title had remained the same for all five years.

And, not least of all, adds Silber, "You also want the look to evolve." Here, again, a designer who knows how to present quantitative data is key.

7. Be on the lookout for new ideas. "Ask yourself what new surveys you can do," he says. Folio did a one-time survey on entry-level salaries. Exploring new aspects of a topic helps keep editors’ perspectives fresh.

Return to the Main Archives Page