COURSE: 641.053 Empirical Research Design
INSTRUCTOR: Manuela Grünangerl
TEAM MEMBERS:
Ekaterina Kartseva, Ehrenreich Leonie
Introduction
This tutorial’s aim is to guide you through the process of designing a Cross-country comparative case study (CNCR). Here, you will find out, whether this approach is suitable for your study, how to conduct a literature review,what to consider at the different stages of your research and see different case study examples, alongside with a CNCR Memo at the end.
Our aim is to show it in a simple and interactive way, so that one could both have a quick look at the tutorial, or go deeper upon the researcher’s need, as well as being able to refer to the real case study examples. We wish you pleasant reading and hope that this tutorial would be helpful for your study!
BEFORE THE RESEARCH
Prepare for the research
CAN INCLUDE:
1. Dominance of single-country studies
2. Ethos
3. Cross-check on the data obtained in interview (Simons, 2009, p.55)
4. Lack of expansive cross-country investigations (Bene, 2023, p. 1711)
5. Process
Structured → Unstructured / In-field setting complete participant → Non-participating OBSERVATION
Formal Observation:
Sense of setting
Rich description
Norms and values in cultures (Simons, 2009, p.55)
Reasons for observing: Informal structures / Body language
EXAMPLE: Develop a codebook, […] where the cross-country reliability test show satisfying coefficients (as conducted in Tonnesen et al., 2023, p. 11)
EXAMPLE: Define dependent variables (numbers of “angry” and “love” reactions from the automatedly collected meta, as done in Tonnesen et al., 2023, p. 10)
LITERATURE REVIEW
Theoretical basis
1) Integrating what others have done / said
2) Criticising previous works
3) Building bridges between topics
4) Identifying central issues in a field
(Cooper, 2010, as cited in Cresswell, 2018, p. 67)
Having ground truth and thick description when using exhaust data can help the researcher understand the social phenomena at play (Bjerre-Nielsen & Glavind, 2022, p. 3)
GROUND TRUTH: A direct observation that can confirm / reject that the sensory data (and the researchers’ interpretation) is correct, i.e. whether the data measures what was assumed.
= > There is a greater need to understand what the exhaust data actually measures—the need for a “ground truth” < =
EXAMPLE: “We find ground truth to be a good metaphor for how human observations can help validate exhaust data because the data quality might not be perfect or may measure something unintended.” (Bjerre-Nielsen & Glavind, 2022, p.2)
THICK DESCRIPTION: Understanding of the social context and mechanisms at play
EXAMPLE: “Illustrated by the classic example from Geertz (1973), about the difference between a twitch and a wink. We might observe two boys doing exactly the same rapid contraction of the right eyelid. For one, the twitch is an involuntary result of a physical impairment, while for the other, it is a conspiratorial signal to a friend. Though these two situations look exactly alike, they are very different but a thick description can distinguish between them.” (as cited in Bjerre-Nielsen & Glavind, 2022, p. 3)
Distinguish between PRIMARY DATA (collected for the specific research question) and SECONDARY DATA (collected for another purpose and subsequently made available to others to reuse and analyze)
(Kitchin, 2014, as cited in Bjerre-Nielsen & Glavind, 2022, p. 2)
RESEARCH DESIGN
Practical advice
• Do not generate too many questions, one, two or at the most three
• Open questions, providing a frame, act as a reminder of focus
(Simons, 2009, p. 31)
Qualitative | Quantitative |
1) Experiments/quasi-experiments 2) Surveys 3) Causal modelling 4) Cost–benefit analysis | Three qualitative methods often used to facilitate in-depth analysis and understanding: 1) Interview 2) Observation 3) Document analysis |
MEASURES: Small-scale surveys, patterns of examination results, questionnaires, descriptive statistics, content analysis (Simons, 2009, p.34) | MEASURES: Critical incidents, open letters, discourse analysis, narratives, video analysis, photographs, log entries, artefacts, case studies, open-ended interviews, observations, document reviews (Simons, 2009, p.55) |
Mixed Method | Action oriented |
1) Surveys | |
2) Interviews | |
3) Observations | Qualitative methods /Stakeholder participation / Interpretative analysis |
4) Document analysis | |
5) Panel review (Simons, 2009) |
• Ethnographic data + Quantitative data and data science techniques → useful inputs that complement other parts of the qualitative analysis. In particular, they can be helpful in the early stages of fieldwork, as a guide to finding patterns that deserve further in-field exploration (Bjerre-Nielsen & Glavind, 2022, p. 4)
• Ethnographic data + models of specific outcomes (machine learning tools)
EXAMPLE: Bjerre-Nielsen et al. (2021) show predictions made from task-specific information (e.g. using high-school grades when predicting college grades) can outperform predictions made using fined grained individual big data. It is conceivable that ethnographers can supply such task-specific data and thereby enhance prediction. (Bjerre-Nielsen & Glavind, 2022, p. 4)
Consider using grounding frameworks:
• Political / Cultural ideologies:
Authoritarianism / Libertanism / Communitarism / Individualism / Combination of the any of the above
• Communicative approaches:
Transactional approach to the publuc / Dialogical approach to the public / Open systems approach to data work
• Governing rationales:
Democratic values / Market capitalism / Privacy protection / Efficency / Pro-innovation
• Governing actors:
Governmental actors / Non-governmental actors / Corporate actors
(Liu and Wang, 2022, p. 3)
• What should be considered + subject to what sense of governance and regulation + who has the right to govern data work, and for whose interest (Liu and Wang, 2022, p. 4)
• When applying quantitative tools to data from ethnographic fieldwork – ethnographic data needs to be structured to some degree and preferably digitized, depending on the format of the data and the application (Glavind and Bjerre-Nielsen, 2021, as cited in Bjerre-Nielsen & Glavind, 2022, p. 4)
RESEARCH
Practical advice
• Ensure homogeneity during the data sampling
HOW? Setting limitations
EXAMPLE: “To ensure the relative homogeneity of the cross-country sample, the investigation is limited to Facebook, which is the dominant social media platform in each member state” (Newman et al., 2020, Teperik et al., 2018, as cited in Bene, 2023, p. 1712)
• Consider country-speficic aspects
Specific domain of social life: Pertinent behaviors / Attitudes of the citizens to be regulated, (H.-J. Andreß et al, 2019, p.17)
NB! The nation is seen as a context in which citizens are embedded. But, most of the time, nations are entities that are remote from the lives of their citizens, and they differ in many ways. (Andreß et. al, 2019)
• Consider how you are going to perform the reliability tests and tackle the language barrier
EXAMPLE: “Due to the national elections taking place at different times and the coding materials being in different languages, cross-country reliability tests Tonnesen et al. (2023, p. 11) could not perform these tests.
4 approaches (by H.-J. Andreß et al., 2019):
1) Analyses of Aggregate data (beware of the composition bias)
Compare national aggregates: Means / Proportions / Correlations / Regression Coefficients across countries, mostly along a descriptive approach (p. 7)
2) Two-Step Analyses
Country estimates of: Means / proportions / correlations / regression coefficients as dependent variables in regression models with country characteristics as explanatory variables (p. 8)
3) Analyses of Disaggregated Data
Disaggregate the contextual information to the lower level and treat these macro- (and meso-)data as if they were microdata. (p. 8)
4) Mixed Effects Analyses
Take the hierarchical nature of the data into account, and estimate individual and contextual effects simultaneously (p. 8)
Type of variable | Macrodata | Mesodata | Microdata |
Genuine Aggregated | Type of political regime (federal vs. unitary) Gross domestic product | Centralization of sectoral wage bargaining Sectoral unemployment rate | Personal political attitudes Total personal income |
• Draft a discussion section, where you discuss the implications of the results
• STEPS: Summarising → Comparing to the literature → Discussing a personal view → Stating limitations and further research (Cresswell, 2018, p. 220)
Weaknesses of CNCR:
1) Impossible determine the direction of causality inherent in the correlations
2) CNCR does not control for time-constant unobserved heterogeneity – may cause bias (Andreß et al., 2019, p.13)
3) Data availability (Boomgaarden and Song, 2019)
4) The problem of having only a small number of higher-level units; and issues of omitted variable bias (Schmidt-Catran et al., 2019, p. 100)
5) Self-selection or convenience sampling in the international surveys (Ebbinghaus, 2005, pp. 133 – 152, as cited in Schmidt-Catran et al., 2019)
6) Small number of included countries in the international surveys (Schmidt-Catran et al., 2019)
7) Planning fieldwork in a foreign country or experiencing a language barrier (Andreß et al., 2019, p.19)
• Adhere to the GDPR standards
• According to Schmidt-Catran et al. (2019) Certain cultures / countries are overrepresented in the social science research
CASE STUDY EXAMPLE
Qualitative design: Ethnography & Critical Analysis
(DeJaeghere et al., 2016, p.70)
Part 1: Two months, in-country fieldwork
⦿ In-person interviews
⦿ Participant observation
⦿ Document analysis
⦿ Students’ journals
Part 2:
⦿ Two months of data collection from the United States
⦿ Document analysis
⦿ Cooperative journaling, phone interviews and Facebook conversations with students to address findings from in the field
In this study the focus was on the English department, which is the biggest student body at Université Cheikh Anta Diop (UCAD) —Sénégal’s largest university. The data that was gathered covered “home region, religion, gender, and ethnicity, as well as contact information and used maximum diversity sampling to identify participants that represented regional, ethnic, religious, and gender diversity” (DeJaeghere et al., 2016, p.70f.)
As the sample five male and five female students were chosen and the researcher conducted “multiple interviews with each of the students for a total of 41 student interviews” (DeJaeghere et al., 2016, p.71). Furthermore, three members of the English faculty were interviewed in order to comprehend how they “perceive agency and if they attempted to cultivate student agency in their classrooms” (DeJaeghere et al., 2016, p.71). Additionally, a completely voluntary survey was conducted that led to 180 completed responses.
CNCR MEMO
• Decide what your study is a case study of and where the boundaries of the case lie, though be aware that these may change in the course of conducting the study.
• Define your initial statement of intent, ‘foreshadowed problems’, issues or research questions but if others arise that seem more appropriate once you are in the field, by all means refocus.
• Keep in mind that case study designs are emergent rather than preordinate. You can respond to the unanticipated, refine the issues and reframe the case.
• Specify the criteria by which you have selected the case; ensure that you can justify your choice in relation to what you can learn about your research topic.
• Identify research questions to frame the study. Do not confuse these with interview questions.
• Decide whether you will start with a theoretical framework or remain open to developing a theory in the process of conducting and analysing the case.
• Choose methods that will offer relevant information to address these questions/issues.
• Think about the possible roles you could adopt as case researcher. Decide which is most suitable for your purpose and audience, but take up others if useful at different stages, as long as they are consistent with agreements made with participants.
• Plan well ahead for your first field visit. Make sure you have piloted your interview/observation schedules, checked your recording equipment, and have plenty of notebooks and pens.
• Stay open to changes in design and methods as understanding in the field grows.
• Draw up schedules of interview questions or issues to be explored; pilot these with colleagues or a small sample within your case.
• Consider what ethical procedures are important to ensure the design is fair to all groups in the case and to establish rapport and trust. See also Case Study Memo 13 in Chapter 6 which comments further on ethical issues to consider in design. (Simons, 2009, p. 38)
SOURCES
For further reading
Andreß, H.-J., Fetchenhauer, D., & Meulemann, H. (2019). Cross-National Comparative Research—Analytical Strategies, Results, and Explanations. KZfSS Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 71(1), 1–28. https://doi.org/10.1007/s11577-019-00594-x
Bene, M. (2023). Who reaps the benefits? A cross-country investigation of the absolute and relative normalization and equalization theses in the 2019 European Parliament elections. New Media and Society, 25(7), 1708–1727. https://doi.org/10.1177/14614448211019688
Boomgaarden, H.G., Song, H. (2019). Media Use and Its Effects in a Cross-National Perspective. Köln Z Soziol 71(1), 545–571. https://doi.org/10.1007/s11577-019-00596-9
Geertz C (1973) Thick description: Towards an interpretive theory of culture. In Geertz, C., (Ed.), The Interpretation of Cultures, 3, (143–168). Basic Books.
Newman N, Fletcher R, Andi S, et al. (2020) Reuters Institute Digital News Report 2020. Oxford: University of Oxford. Available at: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2020-06/DNR_2020_FINAL.pdf
Bjerre-Nielsen, A., & Glavind, K. L. (2022). Ethnographic data in the age of big data: How to compare and combine. Big Data and Society, 9(1). https://doi.org/10.1177/20539517211069893
Boomgaarden, H. G., & Song, H. (2019). Media Use and Its Effects in a Cross-National Perspective. KZfSS Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 71(1), 545–571. https://doi.org/10.1007/s11577-019-00596-9
Cooper, H. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand Oaks, CA: Sage.
DeJaeghere, J., Josić, J., & McCleary, K. (2016). Education and Youth Agency: Qualitative Case Studies in Global Contexts. https://doi.org/10.1007/978-3-319-33344-1
Ebbinghaus, B. (2005). When Less Is More: Selection Problems in Large-N and Small-N Cross-National Comparisons. International Sociology – INT SOCIOL, 20, 133–152. https://doi.org/10.1177/0268580905052366
Kitchin, R. (2014). The Data Revolution: Big Data, Open Data, Data Infrastructures & Their Consequences. SAGE Publications Ltd. https://doi.org/10.4135/9781473909472
Liu, J., & Wang, J. (2022). Social data governance: From reflective practices to comparative synthesis. In Big Data and Society, 9(2). SAGE Publications Ltd. https://doi.org/10.1177/20539517221139786
Schmidt-Catran, A.W., Fairbrother, M. & Andreß, HJ. (2019). Multilevel Models for the Analysis of Comparative Survey Data: Common Problems and Some Solutions. Köln Z Soziol, 71(1), 99–128. https://doi.org/10.1007/s11577-019-00607-9
Simons, H. (2009). Case Study Research in Practice (1st ed.). SAGE Publications, Limited. https://doi.org/10.4135/9781446268322
Teperik D, Senkiv G, Bertolin G, et al. (2018) Virtual Russian world in the Baltics. Available at: https://www.stratcomcoe.org/virtual-russian-world-baltics
Tønnesen, H., Bene, M., Haßler, J., Larsson, A. O., Magin, M., Skogerbø, E., & Wurst, A. K. (2023). Between anger and love: A multi-level study on the impact of policy issues on user reactions in national election campaigns on Facebook in Germany, Hungary, and Norway. New Media and Society. https://doi.org/10.1177/14614448231208122
“Outline” Icons: designed by Freepik.com