E-survey quality ranks higher than print

A recent survey of ASBPE members shows those who solicit reader feedback via e-mail or the Web generally prefer those methods.

By Ira Kerns, Principal Consultant
GuideStar Communications

When conducting a survey, how do ASBPE members develop the project? What methodologies do they use? What is the purpose of the questionnaires? Were members conducting traditional printed surveys or were they conducting them via the Internet?

The answers to these questions are found in the results of an e-survey conducted by GuideStar Communications, New York, N.Y., for ASBPE.

Methodology

Via e-mail, 256 members were invited to participate in the survey. Because of the large number of undeliverable e-mails, the number was reduced to 176. SPSS Exact Tests assured the findings were valid for small samples.

Both an e-mail html attachment questionnaire and a Web site questionnaire on the GuideStar Communications Web site were used to collect the data. Prospective participants received a series of e-mail invitations and reminders during a one-week period.

The questionnaire contained 66 questions relating to members’ use of traditional and electronic survey methods. A response rate of 32% meant 56 members submitted answers.

Given the unusually high percentage of undeliverable e-mail invitations (usually under 5% up to 15% for private populations), this response rate was lower than the response rates for e-surveys GuideStar experiences with employee and customer groups, which are in the 40% to 90% range. However, we find that association response rates to surveys are typically lower than other types of organization survey response rates. The ASBPE response rate is average for a professional society.

Traditional Paper Survey Results

Some 48% of ASBPE members conduct more than four traditional paper surveys a year, with the balance of the membership conducting fewer.

Typically, the surveys have some editorial purpose, including measuring either all readers’, or a subset of all readers’, needs, practices and preferences on a variety of topics (e.g., meeting facilities, travel preferences, use of technology, legislative issues, industry trends, list rankings) and their satisfaction with the magazine. Cited less frequently were advertiser-sponsored surveys and advertising responses.

Data collection methods that members used most are direct mail, with responses mailed (94%) or faxed back (78%). Almost 60% printed questionnaires in their publications, while 45% used surveys inserted into their publication.

Other methods mentioned were direct mail/completed on the web, prepaid phone card survey cards (respondent is given free pre-paid, long-distance phone card to call special number to give answers to questions), and distribution of surveys at industry meetings (collected on-site or mail/faxback).

Survey Responses. Most members (67%) were not satisfied with the effectiveness of traditional data collection, nor with the quality of the responses, with 45%, on average, rating the quality of their survey responses as fair, and 26% rating them as poor. Only 30% rate them as good or very good. No one rated them as excellent. The quality of the responses was even lower when members surveyed sub-sets of their readership. Only 37% calculated a margin of error!

Use of Incentives. Regarding incentives, 47% used them to encourage participation, with 63% saying cash was especially effective. Members also used pre-paid long-distance phone cards, free admission to conferences, business card sweepstakes with gift certificate prizes, $25 charity donations, as well as gifts and cash for a group of 2,000 who participate in surveys all year.

Survey Communications. Less than half, 38%, used pre-survey communications while 48% used stimulation communications (reminders) while conducting surveys. A majority (74%) provided respondents with a description of the results.

Overall Satisfaction. (Thirty-four of 56 people commented.) Most frequently mentioned sources of satisfaction were

1. the strong response rates from core subscribers and

2. the assured quality of the responses.

Less frequently mentioned were

• the reliability of results and assurance of a scientific sample,
• the ability
• to reach all readers, and
• the usefulness of information in planning editorial
• content.

Overall Dissatisfaction. (Forty-eight of 56 people commented.) The most frequently mentioned sources of dissatisfaction were

1) poor response rates overall and from specific segments,

2) declining response rates,

3) time required to collect and analyze the data,

4) expense of printing and mailing survey materials and

5) lack of quality responses on open-end questions (responses too short and not detailed enough).

Mentioned less frequently were

• lack of expertise to compare and analyze data and to determine the best survey design and
• workload to tabulate data by hand and the paperwork involved preparing for surveys.

E-Surveys

Only 29% had conducted an e-survey of which 83% were Web site-based, and 17% were ASCII text pasted in an e-mail message. No one had used an HTML questionnaire attached to an e-mail message.

Members used e-surveys somewhat differently than print surveys. Although readership needs and preferences were an important purpose cited by those using both methods, members used e-surveys more for gathering information to market and sell than print surveys and less for readership profiling and determining readership satisfaction.

Members also mentioned using e-surveys for salary questionnaires, buyer guides and opinion polls.

Member e-surveys, on average, achieved higher response rates (20% vs. 11%), were shorter (15 vs. 26 questions) and were less driven by incentives than their print surveys.

Survey Responses. Overall, members were significantly more satisfied and gave higher ratings to the response rates and quality of responses from their e-surveys than those from their print surveys.

More specifically, 43% were satisfied, and 36% were dissatisfied with the response rates of their e-surveys, compared with 30% ratings of "good or very good" and 67% ratings of "fair or poor" given to their print survey response rates.

Regarding the quality of responses, 64% were satisfied, and 14% were dissatisfied, compared with 30% ratings of "good or very good" and 79% ratings of "fair or poor" given to the quality of their print survey response rates.

Use of Incentives. Far fewer (21%) used incentives on their e-surveys compared with 47% on their print surveys. The primary incentive was cash.

Survey Communications. A majority of members used pre-survey and stimulation communications, provided a summary report, and sent "thank you" messages. Fewer than half implemented these communications on their print surveys except for providing a findings summary (74%).

Advantages of e-surveys. The most frequently mentioned were

1) quicker responses via e-mail,

2) faster results (cuts time from months to weeks),

3) better response rates,

4) lower cost,

5) easier tabulation—responses go directly into an analysis program and

6) easy to cut and paste responses into work documents.

Obstacles to e-surveys. The most frequently mentioned were

1) respondents are self-selecting (difficult to control statistical sample validity),

2) all magazine readers were not on-line,

3) difficulty controlling unqualified or duplicate responses, especially on an open Web site survey,

4) the difficulty in collecting e-mail addresses of those readers who had them,

5) lack of internal resources and e-survey expertise,

6) confidentiality/privacy/ethical concerns,

7) reliability/validity concerns,

8) the ability of members’ e-mail systems to handle a full subscriber survey, and

9) problems relating to the diversity of technological environments.

About the author

Ira Kerns is Principal Consultant at GuideStar Communications, Inc., 305 E. 86th Street, New York, NY 10028. GuideStar is a research firm specializing in high-tech data collection, including e-surveys and tele-surveys. He can be reached by telephone at (212) 426-2333 or by e-mail at gstar1@guidestarco.com. The company’s Web address is www.guidestarco.com.

Return to the Main Archives Page