Recently I began exploring the idea of integrating ChatGPT into SpamLabs in order to build it into an unsolicited message blocker. To do that, the initial idea was to ask ChatGPT in a prompt something in the lines of

is the following message a cold email? respond "yes" or "no" only

The answers were pretty mixed. A lot of false positives (flagging legit messages as cold outreach, for example the "These games from your wishlist are on sale" email sent automatically by Steam will always get marked as cold outreach) or a lot of false negatives (the emails that weren't downright trying to sell you a specific thing by name would not get flagged).

The biggest problem was by far the fact that for the same prompt, I could be getting different results, depending purely on chance. The same message could be flagged as spam once, and then not, and then again, etc.

As a quick and dirty solution I did an ugly "hack" of making the same prompt multiple times, and depending on how many positive results I get, I flag it as spam or not. Worked slightly better, but still not enough (I was getting somewhere like 60% success rate for spam/not spam flagging this way).

Another issue I had was that sometimes ChatGPT was completely ignoring the instruction to only return "yes" or "no", and will start blabbering in a natural language about how the message is or is not spam. I would sometimes getting messages such as "this message is NOT spam" or "it doesn't seem like the message showcases spam behavior" instead of a simple "yes". That was troublesome because parsing the response correctly was the building block of this spam filter. And if it doesn't work, then we would get a lot of wrong results that could (and should) be avoided.

I improved the accuracy and results of the system by doing the following things.

Using the system prompt to severely limit the behavior of ChatGPT

When building a specialized integration, this should be the first thing you always do. Using this, you can impact how the results come out, and also skew the focus of the model towards a certain area, and get less "blabbering" and more "straight to the point" answers.

What is the system prompt? Glad you ask!

The request to the OpenAI completion API (the chat itself) has the following format

{
  "model": "gpt-3.5-turbo",
  "messages": [
    {"role": "system", "content": "you are a helpful assistant"},
    {"user": "user", "content": "which appeared first, the egg or the chicken?"}
  ]
}

And then the response comes according to these prompts. Given that ChatGPT is a statistical language model, it makes sense that the output will be almost always different, if left unchecked and with no restrictions.

Changing the system message content to something more specific, will drastically change how the output is generated. For example, for SpamLabs it can be something like you are a cold email detector, and you will respond with "yes" if the given message is a cold email, or "no" otherwise`

This will change how the bot behaves, and will become more reliable in its responses. And the success rate is slightly better, which is nice. But still not where I want it to be.

Fine-tuning the model

The next step is to create models specialized for the task at hand. I put together a list of about 50 emails that are marked as spam/not spam by me, manually, from my own personal inbox.

Then I used the OpenAI API to create a file with that content, and then create a fine-tuning job with it. The job resulted in a new model after a few minutes of training, and now I got a success rate of like 95% on my data set. Way better than before.

Going forward, the main task regarding this approach would be collecting examples of spam/not spam, compile them in a training set for ChatGPT and update the model to integrate them. Over time, it will become better and better at recognizing unsolicited emails.

Conclusion

ChatGPT, the basic version is definitely fun to use, but pretty limited when it comes to performing specific tasks reliably (the kind of thing you want from a computational system).

By tweaking the system prompt and doing some fine tuning on the model itself can greatly increase the accuracy and improve the behavior of the model to be more in-line with the task at hand.