AI: BBC Editorial Guidelines consultation

Here's our submission on the use of Artificial Intelligence

Our underlying concern is how the BBC ensures it is aware of the nature and scale of AI biases, of the measures taken to mitigate these, and how it as an organisation approaches the pitfalls.

It is vital that safeguards are written into the Editorial Guidelines to ensure that the numerous potential sources of bias are avoided.

LLMs are trained on vast datasets from a variety of sources, including unreliable/non-authoritative sources such as Wikipedia and social media, and naturally reflect the opinions and perspectives present in human language and current mores, which can result in models reproducing these biases.

For example: most global mainstream media has been captured for the last ten years by a belief in gender identity, leading to an identity affirmation bias. The UK and, more slowly, the world, is emerging from an affirmative to a more questioning position: in addition, the gender affirmative position runs counter to certain necessary and empirical truths (you can’t change sex, can’t be born in the wrong body and so on). However an LLM trained on those affirmative sources will replicate the errors. The same will be true on the shifting sands of other controversies.

There is also algorithmic bias stemming from how the algorithm processes and prioritises certain patterns or associations - a challenge that might not be immediately apparent or easy to test. The Editorial Guidelines need to take account of this and recommend approaches such as developing the BBC’s own methods for detecting and auditing bias in training data and the generated content.

This includes human evaluators - the ‘human hand’ - and automated tools to review the outputs for bias across various dimensions, such as race, sex, gender, political alignment, and so on.

However, as well as the training data, such processes themselves may also of course be biased with respect to gender identity belief: to the extent that ‘gender critical’ or sex realist positions are effectively censored by individuals who consider gender affirmation to be of primary importance.

This is a particular risk with US models and sources, given the lack of recognition of gender critical views there, in policy, media and under the law.

If sources like Wikipedia and the texts of law statutes are prioritised as authorities, the problem is compounded since they are, of course, a reproduction of essentially political rather than scientific knowledge. AI models can be fine-tuned on specialised, curated datasets that are designed to reduce

bias. For example, fine-tuning a model on texts that represent diverse viewpoints and counter-stereotypes can help balance the model’s output.

Another example: it is easy to obtain a value judgement about any individual - Was Enoch Powell racist? but when the outputs are tested - eg by challenging the reply ‘Powell believed in hierarchy of races’ - the responses include poor citations and hallucinated quotations. This doesn't mean the output is wrong - in this case, it isn’t - but it does mean the output can't be considered reliable without further investigation. The AI might strongly defend the analysis but be completely unable to provide any actual evidence.

Therefore we urge you to include in the Editorial Guidelines a requirement that adjustments are made in processing that will emphasise factual accuracy. This may seem like an obvious recommendation but in our own field of interest we have seen accuracy abandoned in favour of a political, counter-factual position in every media style guide and almost all output. It is fanciful to imagine that this could never happen with any other issue.

This points to the primacy of Human-in-the-Loop (HITL) Models where AI generated content is reviewed by a human editor or fact-checker - particularly important in politically sensitive contexts where the potential for bias and misinformation can be significant.

We recommend that BBC Editorial Guidelines clearly require:

• bias mitigation frameworks

• an overarching policy governing which use cases of LLMs are legitimate

  and which staff these apply to

• protocols governing the specific ways these these tools can be used, including safeguards to check and validate outputs, and test value statements

• training to ensure all staff involved in content creation are equipped with the tools and knowledge to understand and critically and ethically assess AI-generated content

• compliance records of all sources flagged and verification checks made

• audience-visible tagging of AI-generated content (as part of its misinformation targets)

Our recommendations apply whether the BBC develops its own LLMs (for example - for purposes of political research) or as a potential consumer of AI generated content

*******. End of submission *******