Paper presented to the annual meeting of
the Religious Research Association
Montreal, Quebec
November 6-8, 1998
Ken Bedell
Forum Foundation
4426 Second Avenue, NE
Seattle, WA 98105
kbedell@dnaco.net
Religion and the Internet:
Reflections on Research Strategies
Introduction
Because of the generous support of the Louisville Institute and the sponsorship of both United Methodist Communications and the Centre for the Study of Communication and Culture I was able to study religion on the Internet between July 1997 and April 1998. A report of that research that was specifically prepared for denominational leaders can be found on the Internet at www.religion-research.org/report1.html. In that report I organized the findings of the study around five themes that focused on observations related to religious life and organized religion. In this paper I want to focus on questions related to the process of researching religion and the Internet.
I chose to focus on the research and methodological questions for this paper because I came to believe that it is critical for religious leaders and organizations to figure out what the Internet means for community formation and for individual faith development. Secondly, as I discuss below I came to believe that research on the Internet presents unique challenges to researchers. If researchers are going to take seriously the task of researching the Internet's relationship to religion, then it is important that we take the research methodology questions very serious.
Media studies in general raise questions about methodology. Two important contributions to this discussion have been papers presented by Lynn S. Clark and Alf Linderman last year at the meeting of SSSR/RRA. (Clark 1997 and Linderman 1997) Those papers looked at the qualitative, quantitative and ethnographic research issues in media studies. In this paper I focus specifically on some of the issues of methodology related to researching religion on the Internet.
This paper grows out of reflecting on our experience of trying to study religion on the Internet. In most cases I have tried to use illustrations from our project. However, our project was not focused on how to do research on the Internet. In some cases I can only raise a methodological question or give a subjective report of what seemed to me to be happening. My hope is that this discussion will contribute to and stimulate consideration of the research methods that we use in future research on religion and the Internet.
I will first review research methods that we experimented with.
1) Online questionnaires.
It is quite simple to set up forms on the Internet where people can respond to multiple choice questions or fill in blanks or choose from drop-down lists. Web sites commonly have places for people to register or to leave messages. As a research tool this is a very quick way to obtain data from people who visit a web site.
For our research we prepared a complex questionnaire that included follow-up questions when people indicated particular Internet experience. Rather than telling people to "go to question 7" if they have never bought software over the Internet, we had the computer skip over questions that were irrelevant. We accomplished this by writing a screening set of questions on a first web page and then generating an individualized questionnaire using a cgi program written in the perl. The person filling out the survey experiences a first page of the questionnaire and then a second page. We used a primitive Internet strategy. Java generated questionnaires and new web page tools specifically designed to connect web pages to databases make the process of designing questionnaires on the Internet extremely flexible.
We used forms that generated an e-mail message from each respondent and forms where the data was collected as a database. The advantage of the database process is that we could easily move the data into a spreadsheet or spss for analysis. When the questionnaire results were returned as e-mail messages, they needed to be individually processes. However, for a large project it would probably be possible to write a software program to parse the e-mail messages and compile them into a database.
We also prepared a version of the questionnaire for distribution as an e-mail. People received the questionnaire as an e-mail message and were then asked to reply and fill in their answers. We only received a small number of our total responses from this process. Each response took time to edit since respondents did not give their responses in a uniform way. It may be possible to devise a complex parsing program to read e-mailed questionnaires, but this is probably always going to be like interpreting hand written questionnaires. It is important to look at them to make sure there is an accurate interpretation of the marks made on them.
2) Focus Groups
Focus group research, group interviews or Delphi groups assume a face-to-face setting. In our research we set up two types of mailing lists that served a function similar to focus groups. One group was completely open to anyone who wanted to participate and a second group was by invitation only. In each case the purpose of the mailing list was to collect information from a group of people where everyone's opinion was visible to all participants. Focus groups are most useful when the researcher wants to know how people respond to a particular product or idea. Delphi groups are usually used when the researcher wants to obtain "expert" opinions.
With this research project Internet mailing lists provided a group setting where preliminary results from the research could be discussed and in some cases refined. While it was not a traditional focus group, it served a very similar purpose.
3) Content Analysis
There are areas of activity on the Internet that are public so it is possible to study them using content analysis. We used Folio software to download specific sections of Internet content and then used word search software to identify specific content areas for study. Software usually called BOTS is constantly becoming
more sophisticated for searching the Internet for items that meet certain criteria. Readily available search engines can also be used to search for specific content on the Internet. Although Internet search engines are usually used to help people identify specific content on the Internet, we used them in our research to identify Internet sites in Spanish that have a religious theme. The question we wanted to answer is whether there are important differences between English an Spanish religious sites. Search engines were a powerful tool for finding sites with particular themes.
Difficult to identify populations
While clarity in defining the population that is being studied is a serious issue for most social research, it is a particular problem when studying religion on the Internet. Since a minority of the North American population and a small minority of the global population currently use the Internet, a great deal of caution is necessary when trying to make generalizations about religion and the Internet. Although Internet use is becoming more representative of the general population it is still weighted toward California, youth and men. Because these three categories also are less likely to participate in organized religion, care must be taken in making generalizations from current Internet users.
This can be a particular problem when trying to use a public online questionnaire to collect data. We found this to be a particular problem with our research. Although information about our study was published in both Pittsburgh and Dallas newspapers, reported on Public Radio, and included in popular Internet search engine databases, the respondents to our questionnaire were from a very limited group. They were largely middle aged, mostly male and many were professional religious workers. Although we could draw some conclusions from patters that were found in the way people responded to the survey, it is difficult to generalize from this group to a larger population.
In cases where the researcher wants to know about the people who visit a certain Internet site, then a survey or registration form can be very useful to collect information about those who are attracted to the site. However, usually a researcher will not have such a limited population in mind.
In our project we only e-mailed questionnaires to people that requested them, response rates from a sample that is e-mailed a survey presents the same problems that it does with any survey mailed to a sample. In the United States there is now federal law that governs the use of e-mail to send unsolicited e-mail. While the legalities can certainly be followed, the Internet culture is not very friendly toward unsolicited e-mail.
We also discovered that groups that may appear at first to be homogenous have important distinctions within them. This is illustrated by the results we obtained by asking people what is the most important Internet service for religious denominations to provide. Those who responded to our survey clearly believed that news should be the first priority of denominations in developing Internet strategies. Eighty-one percent listed news as important. Only 65% of the respondents listed the distribution of devotional material as a priority, but 24% listed it as one of the top three most important services that a denomination should provide. So there is a subgroup that likes devotions while the majority puts devotionals very low on the list. Sixty-one percent did not list devotionals as a priority at all.
It may turn out that to look for broad patterns of people using the Internet for religious purposes will turn out to be impossible. Rather than mass movements there may only be subgroups with consistent patterns.
Individuals freedom with the truth on the Internet
Our project provided for 50 follow up telephone interviews. Most, but not all, of the interviews were conducted with people who contacted our web site and indicated on the questionnaire that they would agree to a follow up telephone call.
Researchers constantly struggle to insure that the responses obtained on questionnaires are truthful. In our case the telephone interview provided a way to ask people to confirm what they put onto questionnaires or to explain more fully what they meant by specific responses. The telephone interviews revealed that some people answered the questionnaire by including activities that they hoped to do. But there were also cases where people had quickly filled out the survey but did not provide a complete picture of their own use of the Internet.
My sense is that many people approach the Internet with less care for accuracy than they have when filling out a printed survey questionnaire or when responding to a personal interview. On the Internet, e-mail messages are exchanged where they are quickly sent without making sure that they are carefully worded and exactly accurate. There is an autonomy in mailing lists and chat rooms that allows people to experiment with new ways of presenting themselves. I suspect that this Internet environment made it easier for respondents to our survey to feel comfortable only providing partial information or with claiming to have done things that they planned on doing in the near future. This is an area that I believe demands serious attention as researchers continue to study the Internet. Since questionnaires will probably play a role in Internet research, we need to study if the Internet itself presents a bias into research.
Relationship between culture and the Internet
One of the difficulties of trying to study religion on the Internet is that it is impossible to separate what is happening on the Internet from the cultural transformation that surrounds the Internet.
CNN television illustrates the issue. Each redesign of the CNN format makes the network look more and more like a web page. Or would it be more accurate to say that both CNN television and Internet designers are moving in the same direction. As we look for new patterns of religious expression on the Internet, it is in an environment where patterns of communication and community formation are undergoing transformations. It may be artificial to try to identify specific new patterns of religious expression on the Internet without seeing them in the larger context of changing cultural patterns related to religion.
In our research we thought we could identify an interest on the part of people to participate in response to needs directly without the mediation of religious denominational organizations. In this case the question needs to be asked whether this has anything to do with religion and the Internet. It might be that we were simply observing a changing cultural pattern where people do not see organized religion as a useful vehicle for their learning about or responding to important issues.
Defining meaningful outliers
Particularly for quantitative analysis of data, outliers are usually seen as a problem. My own introduction to this issue was a graduate multiple regression text that presented the difficulty and then concluded, "if outliers are few (less then 1% or 2% of n) and not very extreme, they are probably best left alone." (Cohen and Cohen 1983:128). I cannot find any reference to the issue of outliers in the popular text The Practice of Social Research by Earl Babbie (Babbie, 1995).
Most people who will be doing research on religion and the Internet have been trained to be very suspicious of outliers. . Tukey describes the box-and-whisker in his 1977 book Exploratory Data Analysis. To this day social scientists are trained to use this tool to identify outliers. However, it is interesting to note that the box-and-whisker did not become available to SPSS users until version 9. Another method used in quantitative analysis to identify outliers is to examine a scatter plot of residuals. Since residuals are a comparison of each data point to the expected result, outliers are very evident in a scatter plot.
Frederick Hartwig's presentation on outliers is typical of the training that current social scientists have received and continue to receive. In the Sage series on Quantitative Applications in the Social Sciences he discusses the use of the box-and-whisker plot designed by J. W. Tukey. Hartwig, to his credit, does note that "it may be the outliers themselves on which the analysis should be concentrated. For example, communities with abnormally low crime rates maybe the most instructive ones." (Hartwig, 1998: 28.) However, Hartwig does not discuss how such analysis might proceed or what methods might be used. The real message of his section on outliers is that they indicate a problem with the data. He gives an example of a study where data from Cuba claimed to have a very large number of miles of railroad. He first points out how the outlier has a strong impact on the mean and standard deviation. Then he writes, "One other point can be made with this example, The value for the outlying case, Cuba, is sufficiently different from the other values that one should suspect an error in the data."
Researchers who are studying religion and the Internet have a number of challenges related to the cases that are outliers. We found this to be a particularly difficult problem because there are few ways to clearly determine which cases are eccentric and which are pioneers or early adapters. In our research we were not able to identify any religious practices that have been adopted by a significantly large group of people. The question then became whether any significance should be given to the emergence of new behaviors. Several examples will illustrate the difficulty. Several Christian ministers have attempted to establish online congregations. When we looked into every case that came to our attention we could not discover one where more than a very small number of people (usually fewer than ten) are participating in a regular basis. Clearly these online congregations are not the norm for the average or even a measurable minority of Internet users. The leaders and participants could only be called outliers. The question becomes: are these Internet users important in the study of religion and the Internet because they, as they seem to believe themselves, are discovering and developing forms of religious practice that will be the norm in a few year.
High expectations
In our research we asked people about what they think that religious organizations should be doing related to the Internet. Most people had no difficulty coming up with a list of items. I suspect that there is a certain amount of Gold Rush mentality in current Internet users. I imagine that a researcher talking to people getting ready to travel to California to find gold would find that everyone had an answers to the question: What are you going to buy with the gold that you find? Today people seem to be able to easily answer the question: What could the Internet be used for.
A recent national study asked young people if they expect to use the Internet as part of their spiritual life in the future. Not surprisingly a large number said yes. This reseach should be a warning to any researchers that developing research strategies to learn about religion and the Internet requires very careful planning. Any research that involves trying to determine what people expect or plan on doing is extremely complex. However, with the Internet, there are added complexities because of the high cultural expectation for the Internet.
Summary
In many ways the Internet provides opportunities for research because of the public nature of much of the Internet and the possibility of adapting common research methods such as questionnaires, focus groups and content analysis.
There are also a number of methodological and design issues presented by the Internet. These include identifying groups and populations, obtaining truthful responses over the Internet, distinguishing between Internet and cultural factors, identifying which outliers are useful cases and overcoming the high expectations that people have for the Internet.
All of these issues are well known to research designers. In the case of research on religion and the Internet they may require special solutions.
Conclusions
This discussion has focused on questions of method rather than purpose. However, the issues raised here suggest that research on religion and the Internet is in some ways a circular problem. For example, the extent to which people provide truthful responses to Internet questionnairs is an issue for a researchers who wants to obtain a true picture of what is happening. But this also raises important questions about how people interact with each other and institutions on the Internet. The Nature of community and individual behavior on the Internet become prior questions. It may be that the kinds of questions Marshall McLuhan raised The Gutenberg Galaxy: The making of Typographic Man (McLuhan 1962) need to become the priority for research on religion and the Internet.
Following this line of argument the issues presented by research methodology present in themselves important questions about the nature of the Internet and the nature of the forms of human organization that use the Internet. So the difficulty in clarifying groups in Internet research raises questions about how individuals understand themselves to be part of groups in an Internet environment. When people join and leave discussion groups with a click, what is the nature of commitment or the requirment for social conformity to group norms.
How does self presentation work in an Internet environment is an issue that grows out of questions about truth telling on Internet questionnaires. We might study the way an individual develops a self identity and the role of testing various identities on the internet in the process of personality formation. This naturally leads to important questions about spiritual formation.
The problems of distinguishing between Internet and cultural factors, identifying which outliers are useful cases and overcoming the high expectations that people have for the Internet are all related to coming to a better understanding of the nature of Internet interaction and its role in the emerging cultural patterns.
While religious leaders need quick answers to questions about religion on the Internet that will help them determine how the existing religious organizations should invest in Internet activity, researchers have an important task ahead of developing a strong theoretical understanding upon which to base their research.
References
Babbie, Earl (1995). The Practice of Social Research. Belmont, CA: Wadsworth Publishing.
Clark, Lynn S. (1997). "The Method of Critical Ethnography: Linking Reflexive Empirical Resarch with Critical Theory." Paper presesnted at the SSSR Annual meeting in San Diego, CA, November 7, 1997.
Cohen, Jacob and Patricia Cohen (1983). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence ErlBaum Associates.
Hartwig, Frederick (1979) with Brian E. Dearing. Exploratory Data Analysis. Beverly Hills, CA: SAGE Publications.
Linderman, Alf (1997). "Qualitative vs. Quantitative Methods in the Study of Religious Television in Sweden." Paper presented at the SSSR Annual meeting in San Diego, CA, November 7, 1997.
McLuhan, Marshall (1962). The Gutenberg Galazy: The Making of Typographic Man. Toronto: University of Toronto Press.
Tukey. J.W. (1977). Exploratory Data Analysis. Reading, MA: Addison-Wesley.