• The_Walkening [none/use name]English
    18•2 days ago

    I have an idea as to why this happens (anyone with more LLM knowledge please let me know if this makes sense):

    1. ChatGPT uses the example code to identify other examples of insecure code
    2. Insecure code is found in a corpus of text that contains this sort of language (say, a forum full of racist hackers)
    3. Because LLMs don’t actually know the difference between language and code (in the sense that you’re looking for the code and not the language) or anything else, they’ll return responses similar to the examples in the corpus because it’s trying to return a “best match” based on the fine tuning.

    Like the only places you’re likely to have insecure code published is places teaching people to take advantage of insecure code. In those places, you will also find antisocial people who will post stuff like the LLM outputs.

    • semioticbreakdown [she/her]English
      3•1 day ago

      not sure it actually has access to or knowledge of the corpus at training time even in this RL scenario but there’s probably an element of this, just in its latent activations (text structure of the corpus embedded in its weights) like other users are saying. but it’s important to note that it doesnt identify anything. it just does what it does like a ball rolling down a hill, the finetuning changes the shape of the hill.

      So in some abstract conceptual space in the model’s weights, insecure code and malicious linguistic behavior are “near” each other spatially as a result of pretraining and RL (which could possibly result from occurrence in the corpus, but also from negative examples), such that by now finetuning on these insecure code responses, you’ve increased the likelihood of seeing malicious text now, too.