Online content analysis

Online content analysis or online textual analysis refers to a collection of research techniques used to describe and make inferences about online material through systematic coding and interpretation. Online content analysis is a form of content analysis for analysis of Internet-based communication.

History and definition

Content analysis as a systematic examination and interpretation of communication dates back to at least the 17th century. However, it was not until the rise of the newspaper in the early 20th century that the mass production of printed material created a demand for quantitative analysis of printed words.^[1]

Berelson’s (1952) definition provides an underlying basis for textual analysis as a "research technique for the objective, systematic and quantitative description of the manifest content of communication."^[2] Content analysis consists of categorizing units of texts (i.e. sentences, quasi-sentences, paragraphs, documents, web pages, etc.) according to their substantive characteristics in order to construct a dataset that allows the analyst to interpret texts and draw inferences. While content analysis is often quantitative, researchers conceptualize the technique as inherently mixed methods because textual coding requires a high degree of qualitative interpretation.^[3] Social scientists have used this technique to investigate research questions concerning mass media,^[1] media effects^[4] and agenda setting.^[5]

With the rise of online communication, content analysis techniques have been adapted and applied to internet research. As with the rise of newspapers, the proliferation of online content provides an expanded opportunity for researchers interested in content analysis. While the use of online sources presents new research problems and opportunities, the basic research procedure of online content analysis outlined by McMillan (2000) is virtually indistinguishable from content analysis using offline sources:

1. Formulate a research question with a focus on identifying testable hypotheses that may lead to theoretical advancements.
2. Define a sampling frame that a sample will be drawn from, and construct a sample (often called a ‘corpus’) of content to be analyzed.
3. Develop and implement a coding scheme that can be used to categorize content in order to answer the question identified in step 1. This necessitates specifying a time period, a context unit in which content is embedded, and a coding unit which categorizes the content.
4. Train coders to consistently implement the coding scheme and verify reliability among coders. This is a key step in ensuring replicability of the analysis.
5. Analyze and interpret the data. Test hypotheses advanced in step 1 and draw conclusions about the content represented in the dataset.

Content analysis in internet research

Since the rise of online communication, scholars have discussed how to adapt textual analysis techniques to study web-based content. The nature of online sources necessitates particular care in many of the steps of a content analysis compared to offline sources.

While offline content such as printed text remains static once produced, online content can frequently change. The dynamic nature of online material combined with the large and increasing volume of online content can make it challenging to construct a sampling frame from which to draw a random sample. The content of a site may also differ across users, requiring careful specification of the sampling frame. Some researchers have used search engines to construct sampling frames. This technique has disadvantages because search engine results are unsystematic and non-random making them unreliable for obtaining an unbiased sample. The sampling frame issue can be circumvented by using an entire population of interest, such as tweets by particular Twitter users^[6] or online archived content of certain newspapers as the sampling frame.^[7] Changes to online material can make categorizing content (step 3) more challenging. Because online content can change frequently it is particularly important to note the time period over which the sample is collected. A useful step is to archive the sample contehnt in order to prevent changes from being made.

Online content is also non-linear. Printed text has clearly delineated boundaries that can be used to identify context units (e.g., a newspaper article). The bounds of online content to be used in a sample are less easily defined. Early online content analysts often specified a ‘Web site’ as a context unit, without a clear definition of what they meant.^[2] Researchers recommend clearly and consistently defining what a ‘web page’ consists of, or reducing the size of the context unit to a feature on a website.^[2][3] Researchers have also made use of more discrete units of online communication such as web comments^[8] or tweets.^[6]

Automatic content analysis

The rise of online content has dramatically increased the amount of digital text that can be used in research. The quantity of text available has motivated methodological innovations in order to make sense of textual datasets that are too large to be practically hand-coded as had been the conventional methodological practice.^[3][7] Advances in methodology together with the increasing capacity and decreasing expense of computation has allowed researchers to use techniques that were previously unavailable to analyze large sets of textual content.

Automatic content analysis represents a slight departure from McMillan's online content analysis procedure in that human coders are being supplemented by a computational method, and some of these methods do not require categories to be defined in advanced. Quantitative textual analysis models often employ 'bag of words' methods that remove word ordering, delete words that are very common and very uncommon, and simplify words through lemmatisation or stemming that reduces the dimensionality of the text by reducing complex words to their root word.^[9] While these methods are fundamentally reductionist in the way they interpret text, they can be very useful if they are correctly applied and validated.

Grimmer and Stewart (2013) identify two main categories of automatic textual analysis: supervised and unsupervised methods. Supervised methods involve creating a coding scheme and manually coding a sub-sample of the documents that the researcher wants to analyze. Ideally, the sub-sample, called a 'training set' is representative of the sample as a whole. The coded training set is then used to 'teach' an algorithm the how the words in the documents correspond to each coding category. The algorithm can be applied to automatically analyze the remained of the documents in the corpus.^[9]

• Dictionary Methods: the researcher pre-selects a set of keywords (n-gram) for each category. The machine then uses these keywords to classify each text unit into a category.
• Individual Methods: the researcher pre-labels a sample of texts and trains a machine-learning algorithm (i.e. SVM algorithm) using those labels. The machine labels the remainder of the observations by extrapolating information from the training set.
• Ensemble Methods: instead of using only one machine-learning algorithm, the researcher trains a set of them and uses the resulting multiple labels to label the rest of the observations (see Collingwood and Wiklerson 2011 for more details).^[10]
• Supervised Ideological Scaling (i.e. wordscores) is used to place different text units along an ideological continuum. The researcher selects two sets of texts that represent each ideological extreme, which the algorithm can use to identify words that belong to each extreme point. The remainder of the texts in the corpus are scaled depending on how many words of each extreme reference they contain.^[11]

Unsupervised methods can be used when a set of categories for coding cannot be well-defined prior to analysis. Unlike supervised methods, human coders are not required to train the algorithm. One key choice for researchers when applying unsupervised methods is selecting the number of categories to sort documents into rather than defining what the categories are in advance.

• Single membership models: these models automatically cluster texts into different categories that are mutually exclusive, and documents are coded into one and only one category. As pointed out by Grimmer and Stewart (16), "each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes and ideal clustering; and (3) an optimization algorithm."^[9]
• Mixed membership models: According also to Grimmer and Stewart (17), mixed membership models "improve the output of single-membership models by including additional and problem-specific structure."^[9] Mixed membership FAC models classifies individual words within each document into categories, allowing the document as a whole to be a part of multiple categories simultaneously. Topic models represent one example of mixed membership FAC that can be used to analyze changes in focus of political actors^[6] or newspaper articles.^[7] One of the most used topic modeling technique is LDA.
• Unsupervised Ideological Scaling (i.e. wordsfish): algorithms that allocate text units into an ideological continuum depending on shared grammatical content. Contrary to supervised scaling methods such as wordscores, methods such as wordfish^[12] do not require that the researcher provides samples of extreme ideological texts.

Validation

Results of supervised methods can be validated by drawing a distinct sub-sample of the corpus, called a 'validation set'. Documents in the validation set can be hand-coded and compared to the automatic coding output to evaluate how well the algorithm replicated human coding. This comparison can take the form of inter-coder reliability scores like those used to validate the consistency of human coders in traditional textual analysis.

Validation of unsupervised methods can be carried out in several ways.

• Semantic (or internal) validity represents how well documents in each identified cluster represent a distinct, categorical unit. In a topic model, this would be the extent to which the documents in each cluster represent the same topic. This can be tested by creating a validation set that human coders use to manually validate topic choice or the relatedness of within-cluster documents compared to documents from different clusters.
• Predictive (or external) validity is the extent to which shifts in the frequency of each cluster can be explained by external events. If clusters of topics are valid, the topics that are most prominent should respond across time in a predictable way as a result of outside events that occur.

Challenges in online textual analysis

Despite the continuous evolution of text-analysis in the social science, there are still some unsolved methodological concerns. This is a (non-exclusive) list with some of this concerns:

• When should researchers define their categories? Ex-ante, back-and-forth, or ad-hoc? Some social scientists argue that researchers should build their theory, expectations and methods (in this case specific categories they will use to classify different text units) before they start collecting and studying the data^[13] whereas some others support that defining a set of categories is a back-and-forth process.^[14][15]
• Validation. Although most researchers report validation measurements for their methods (i.e. inter-coder reliability, precision and recall estimates, confusion matrices, etc.), some others do not. In particular, a larger number of academics are concerned about how some topic modeling techniques can hardly be validated.^[16]
• Random Samples. On the one hand, it is extremely hard to know how many units of one type of texts (for example blogposts) are in a certain time in the Internet. Thus, since most of the time the universe is unknown, how can researcher select a random sample? If in some cases is almost impossible to get a random sample, should researchers work with samples or should they try to collect all the text units that they observer? And on the other hand, sometimes researchers have to work with samples that are given to them by some search engines (i.e. Google) and online companies (i.e. Twitter) but the research do not have access to how these samples have been generated and whether they are random or not. Should researches use such samples?

See also

• Content analysis
• Text mining

References

1. Krippendorff, Klaus (2012). Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
2. McMillan, Sally J. (March 2000). "The Microscope and the Moving Target: The Challenge of Applying Content Analysis to the World Wide Web". Journalism and Mass Communication Quarterly. 77 (1): 80–98. doi:10.1177/107769900007700107.
3. van Selm, Martine; Jankowski, Nick (2005). Content Analysis of Internet-Based Documents. Unpublished Manuscript.
4. Riffe, Daniel; Lacy, Stephen; Fico, Frederick (1998). Analyzing Media Messages: Using Quantitative Content Analysis in Research. Mahwah, New Jersey, London: Lawrence Erlbaum.
5. Baumgartner, Frank; Jones, Bryan (1993). Agendas and Instability in American Politics. Chicago. University of Chicao Press.
6. Barberá, Pablo; Bonneau, Richard; Egan, Patrick; Jost, John; Nagler, Jonathan; Tucker, Joshua (2014). "Leaders or Followers? Measuring Political Responsiveness in the U.S. Congress Using Social Media Data.". Prepared for delivery at the Annual Meeting of the American Political Science Association,.
7. DiMaggio, Paul; Nag, Manish; Blei, David (December 2013). "Exploiting affinities between topic modeling and the sociological perspective on culture: Application to newspaper coverage of U.S. government arts funding". Poetics. 41 (6): 570–606.
8. Mishne, Gilad; Glance, Natalie (2006). "Leave a reply: An analysis of weblog comments". Third Annual Conference on the Weblogging Ecosystem.
9. Grimmer, Justin; Stewart, Brandon (2013). "Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts". Political Analysis. 21 (3): 1–31.
10. Collingwood, Loren and John Wilkerson. (2011). Tradeoffs in Accuracy and Efficiency in supervised Learning Methods, in The Journal of Information Technology and Politics, Paper 4.
11. Gerber, Elisabeth, and Jeff Lewis. 2004. Beyond the median: Voter preferences, district heterogeneity, and political representation. Journal of Political Economy 112(6):1364–83.
12. Slapin, Jonathan, and Sven-Oliver Proksch. 2008. A scaling model for estimating time-series party positions from texts. American Journal of Political Science 52(3):705–22.
13. King, Gary, Robert O. Keohane, & Sidney Verba. (1994). Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton: Prince University Press.
14. Herring, Susan C. (2009). Hunsinger, Jeremy, ed. Web Content Analysis: Expanding the Paradigm. Springer Netherlands. pp. 233–249. ISBN 978-1-4020-9788-1. Retrieved 2015-04-11.
15. Saldana Johnny. (2009). The Coding Manual for Qualitative Research. London: SAGE Publication Ltd.
16. Chuang, Jason, John D. Wilkerson, Rebecca Weiss, Dustin Tingley, Brandon M. Stewart, Margaret E. Roberts, Forough Poursabzi-Sangdeh, Justin Grimmer, Leah Findlater, Jordan Boyd-Graber, and Jeffrey Heer. (2014). Computer-Assisted Content Analysis: Topic Models for Exploring Multiple Subjective Interpretations. Paper presented at the Conference on Neural Information Processing Systems (NIPS). Workshop on HumanPropelled Machine Learning. Montreal, Canada.