Why the BBC owes a real apology to The Better India

The BBC has annoyed quite a few people, especially among the Right-wing, ever since it published its recent research paper titled “Duty, Identity, Credibility: ‘Fake News’ and the ordinary citizen in India.”

The report of the research it conducted was initially published as an exhaustive 105-page document, created after “participants gave the BBC extensive access to their phones over a seven-day period, allowing the researchers to examine the kinds of material they shared, whom they shared it with and how often.” The research found that “people in India are sharing fake news stories with nationalistic messages for ‘nation building’ purposes, with consolidation of national identity taking precedence over the need to fact-check a story.”

The initial 105-page report was removed from the BBC’s website last weekend after questions were raised about the methodology adopted by them. The document was uploaded back on Tuesday evening, with an added section titled “Further Information on Methodology.”

Nothing has been deleted or amended in the “updated” report. Apart from an additional section about the methodology being added, the BBC also mentioned that they had mistranslated the term “diwaliya” and had corrected it in their updated document. They also stated that the three fact-checking initiatives that have been used to arrive at sources “known to have published fake news” are now signposted in multiple places in the report and that more clarity has been introduced in its network analysis descriptions, maps, and labels.

It goes unsaid that such pulling down, updating, and re-clarifying of its methodology, does not portray the news portal in a great light. For a report that makes such heavy-duty claims, one would have expected the BBC to do a more thorough job. By temporarily pulling down its report so as to clarify its stance on the methodology used by them, it (BBC) has already invited the wrath of one such Right-wing publication, which has pointed out “20 instances of fake news that were furthered by the BBC.” In the past week, about eight such stories have been published on the news portal, which—ironically enough— is one of whose Twitter handles has been listed under the section “known to have published fake news/more in the pro-BJP cluster” in the BBC report.

But what seems to be even more damaging for BBC’s reputation is its adamant and unabashed refusal to render a public apology to The Better India, a news outlet wrongly accused by them of sharing Fake News. When The Better India wrote to them, Sanatanu Chakrabarti, the BBC research’s lead author, confirmed their mistake in a private email, and apologised for committing a“human error.” The email further stated: “The Better India should not have been included and we would like to apologise to you for this mistake.” However, The Better India expected “a public apology from them for maligning years of our hard work.”

Now, even after going through all the trouble of temporarily pulling down its report so that it could be updated with an entirely new section, the BBC is yet to issue a public apology. No such apology has been issued even from its primary Twitter handles, let alone on its new portal. The public apology was even missing from the “updated” document that was uploaded on Tuesday. In its “Amendments, Corrections, and Additions” section, the new report stated that “an online publisher—The Better India—that was incorrectly labelled on the Twitter network map “has been removed.”

Dhimant Parekh, Founder, The Better India, says the “obscure statement” made by BBC in its updated report will not cut it; a public apology from the official BBC Twitter handle is what is the need of the hour. “We want a public apology from the official BBC Twitter handle because that’s the handle that distributed the report first, which means that their followers got to consume the report from there. What they (BBC) ended up doing so is apologising to us over email. Then, BBC’s Rupa Jha and Trushar Barot also replied on my Twitter thread confirming that they had made a mistake, but their following (on Twitter) is not the same following like that of the BBC.”

Referring to the updated report, Parekh said: “They’ve made some obscure statement about The Better India having been removed from Twitter network map. We are very clear that public apology will have to come officially from BBC. If it doesn’t happen, we will consider our other options—I don’t think we want to let this go. I just wish that the BBC would actually step up. They are questioning the integrity of a lot of media reports, but the very fact that they have committed a human error and then not apologised for it, makes them look like an entity that is not ethical at all. For an unethical entity to write about who is spreading fake news, doesn’t make any sense to me. How can you firstly accuse someone of spreading fake news and then say that it was a human error—when the whole crux of the matter was to point out to others who are spreading Fake News? As a small-time publisher, it would be a lot for us to go legal. All we are saying: let the people who have consumed your first report know that this is false.”

This is definitely something that the broadcasting house should have been more responsible about, and also one of the few reasons responsible for turning the entire report into an episode of “Ripley’s Believe It Or Not.” 

The  question of sample size

The BBC has always maintained that their report is an “ethnographic” one—even though they might not have gone out of their way to explain to readers what the salient features of an ethnographic research really are.

An ethnographic approach is a type of qualitative research that “typically requires a smaller sample size than qualitative analysis” and is also primarily based on “participant observation”—two boxes that the BBC seems to have checked off with its 120+ hours of in-depth interviews with 40 respondents. It (ethnography) is a study of the culture and social organisation of a particular community, wherein the goal of the researchers is to achieve a point of saturation. “Saturation occurs when adding more participants to the study does not result in additional perspectives or information” and it is this concept of saturation that is recommended by Glaser and Strauss for achieving an appropriate sample size when it comes to qualitative studies.

But in the 120+ hours of in-depth interviews, the BBC conducted with its respondents, it remains unclear whether saturation point was ever reached or not.

The broadcasting agency also received major flak for the sample size it used in determining the results and findings of its research. Its total sample size for India was 40, comprising 10 people each from across four different age categories. Over 120+ hours of interviews were conducted with all these respondents. Analysis of 16,000 Twitter profiles, 3,200 Facebook pages, and the scanning of 47,543 reports on Fake News was also carried out as a part of the research.

However, it wouldn’t have hurt them (BBC) to make it explicitly clear—at the very outset—of the relationship between an ethnographic research and the reason why using such a lowballing sample size was acceptable while adopting this method. Considering the fact that the representative sample size of a country with 120 million people has been narrowed down to only 40 individuals in its report, the BBC could have made its methodology amply transparent since the beginning, instead of waiting for others to pose questions about it.  

Moreover, five out of seven times the word “ethnographic” was used in the BBC report of Fake News, it was preceded either by “semi” or “auto.” What does that even mean?

Newslaundry reached out to five experts in an effort to understand why such a small sample size was chosen in the first place, and whether this is the norm in a qualitative research.

Dr B.K Sharma, Ex-Director of the government-run Directorate of Economics and Statistics in New Delhi, said that the main difference between a qualitative and quantitative research/survey method is that the former deals mainly with understanding people’s perceptions, while the latter deals with things that can be quantified in terms of numbers. “Your perception or views cannot be quantified,” he said. “During a qualitative research, the researcher asks about your views on different things such as the economy, public policy etc, and then a research is conducted based on the individual perception of different individuals. When you cannot quantify something, you move to examine its qualitative aspect.”

However, Sharma said a sample size of 40 was “extremely small.” “Most of the research surveys being done today are on a small scale. In this case, the 40 respondents could have been from 40 different backgrounds, thereby increasing the diversity range of the sample size. What is most important is the kind of questions asked by the researcher.”

According to Sunil Kumar Sinha, Director and Principal Economist, India Ratings and Research Limited, the sample size of 40 respondents is simply too small to draw conclusions for a nation-wide trend. “No matter what you do, these 40 people cannot be representative of 120 crores,” he said. “It is statistically irrelevant.”

Professor Arun Kumar, an ex-JNU professor who is currently the Malcolm Adiseshih Chair Professor at the Institute of Social Sciences, also agreed with Sinha and Sharma when it came to the lowball sample size adopted by the BBC. “In statistics, the reliability of your conclusion depends mainly on your sample size. 40 people would definitely not be able to represent the entire country as a whole; it is more of an indication, not something you can say is a rule.” He also pointed out that an “ethnographic” study means that “you do a very limited study and on that basis, you draw some conclusion but you know it cannot be generalised.”

However, Dr Tulsi Jayakumar, Professor of Economics and the Chairperson of the Family Managed Business at the S.P. Jain Institute of Management & Research in Mumbai, tends to disagree. “This type of approach (qualitative/ethnographic) would be more robust than a quantitative one,” she said, adding that “design thinking” helps in finding solutions to problems that one may not get by implying traditional quantitative data. “Sample size does not matter—it is the quality of the sample that should matter.”

Dr Manindra Thakur, who has been teaching research methods for the last decade at the Centre for Political Studies, Jawaharlal Nehru University, also weighs in on Jayakumar’s side. “There is no fixed sampling for ethnography,” he said. “Ethnography relies more on how do you choose the sample and what kind of a sample have you taken. The basic difference between qualitative and quantitative research methods are that in the latter, sampling should be random i.e. anyone should have a chance to fall into your sample. Your numbers should be sufficiently high to suggest general trends. But in ethnography, your sample includes people who are actually undergoing/involved in a particular process. More than the size of the sample, it is the right quality of sample that is important in ethnography.”

All the experts Newslaundry spoke with were unable to pinpoint what “semi-ethnography” or “auto-ethnography” means.

Newslaundry has also written to the BBC asking why has a public apology not been issued to The Better India, and what exactly is the difference between a “semi-ethnographic” research and “auto-ethnography.”

In the response we received, it was stated that “the BBC’s apology to The Better India is already public.” “We wrote to them directly and they made our apology public on Twitter. In addition, the Editor of BBC Indian Languages, Rupa Jha, confirmed that we had made a mistake and apologised to The Better India from her official account.”

The response also clarified what “semi-ethnographic” and “auto-ethnographic” means. “The whole study uses ethnographic approaches, meaning that in-depth interviews were conducted on location with respondents. The terms ‘semi-ethnographic’ and ‘auto-ethnographic’ simply refer to the way information was collected ( ‘semi’ refers to the duration of contact with respondents and ‘auto’ relates to data that respondents themselves collected and shared).

For now, the BBC has maintained that “The research report states clearly that this project is exploratory in nature. It is intended to serve as a ‘starting point in the research conversation’—not the final word”. By now, that’s amply clear.