Why is AI So Predictable? The Random Number Test
The Experiment: Ask the three most used general-purpose chatbots to pick a random number between 1-30, and they will mostly give you the same answer. Those three being: 1. ChatGPT 2. Google Gemini 3. Anthropic's Claude
In my testing, the results in the beginning were almost identical (and it works for almost every common AI bot): 1. First given number is 17 2. Second given number is 23 The third number is often 7 or 8, although it varies more the more prompts you send within the same chat.
A truly random pick from 1 to 30 would result in each number having a roughly 3.3% chance of being picked; so this repetition shows a pattern that the AI doesn't just "follow your instruction". Some type of pattern is happening.
There are two predictable explanations for this; although we'll start with the bare minimum.
Reason 1 (and a fairly popular guess): Prompt Caching Usually, people's first idea almost always lands on prompt caching, including mine. It's a real thing, and ChatGPT, Gemini, and Claude all use it in some way, shape, or form, although it does not explain repeated answers, especially accross all chatbots.
Prompt Caching is a "speed and cost feature". Antrhopic, for example, describes it as letting the model resume from a "saved prefix" of your prompt, which cuts the time it takes to answer your question, and the cost for tasks that are constantly repeated, or prompts that are simply consistent with eachother (https://www.promptlayer.com/glossary/prompt-caching/).
However, Prompt Caching, in general, catches the question, not the actual answer that's being outputted. The model generates a fairly fresh response every time, so usually a cache won't give you the same number twice (although it does have some general influence with this specific task).
A shorter prompt like this, however, hardly qualifies as well; Anthropic's cahce for example lasts for 5 minutes by default (https://trigger.dev/docs/ai-chat/prompt-caching).
And, finally, it doesn't explain the agreement that multiple chatbots share. If caching was somehow the cause, it is very rare that those answers would be the same across multiple different chatbots.
So this can't be it.
Reason 2: AI is a Language Model; They are trained on us, and language.
This is a super strong explanation, and has a lot of evidence behind it. A large language model doesn't have a random dice that simply rolls itself; it predicts the text a person would most likely type next, based on a huge amount of human writing. It is trying to predict what a person would say; not come up with it on it's own. This is why you will often here people consider AI as plagarism; it is plagarizing off all the writing and language that it has been trained on. When you ask it for a 'random' number, it is not just generating a number; it is giving you a number people would most often give when they are trying to sound random, or when you would ask them the same question.
Humans are extremely poor at being random, and at many times we can be very predictable. I sat down, and thought about it myself: How would we, as humans, pick a random number between 1 and 30?
1. Avoid the Ends; It's dumb to pick 1, 2, 29, or 30. So that leaves roughly 6 to 24 if avoiding anything too close to 1 or 30.
2. Avoid Numbers that look 'too perfect'; multiplies of 5 and 10 (10, 15, 20), repeated digits (11, 22) feel patterned so those are bad choices, etc. This leaves 6, 7, 8, 9, 12, 13, 14, 16, 17, 18, 19, 21, 23, and 24.
3. Generally avoid single digits; They feel way too small and way too easy; this removes 6-9. Since the majority of the numbers are also double digits, it feels more random to not pick a single digit.
4. Avoiding Numbers that Carry Baggage, or in general are popular. 21 is blackjack, 24 is a super common multiple of 6 and it's also the total numbers of hours in a day, and they feel chosen rather than being super random.
5. (the biggest influence in my opinion --) Lean towards prime numbers. Primes feel super irregular, unpredictable, and we hardly ever see them in the real number. Among what currently remains (12, 13, 14, 16, 17, 18, 19, 23), the primes are 13, 17, 19, and 23.
The two most common AI answers, 17 and 23, are both in that final group of numbers. While it's not quite perfect, with 13 being famously unlucky (step 4), and some other minor obscurities, there is a super clear pattern. AI is treating these steps as esentially models of human intelligence and intuition; not a formula to pick a truly random number. It is trained on humans.
The next step would to go to bigger ranges of numbers. For example, when humans were asked for a number from 1 to 100, they picked 37 more than any other number.
Data scientist Colon Fraser also asked ChatGPT for a number from 1 to 100 in 2,000 seperate, individual chats, and roughly TEN PERCENT of the answers were 42, which was described as the "answer to everything" (https://futurism.com/the-byte/chatgpt-random-numbers).
These change periodically as AI releases new models, trained on new data. Now, ChatGPT is super likely to lean towards 47, and Gemini is super likely to take a hit for 72. However, a super clear pattern across all of these chatbots is that they love avoiding round numbers, repeated digits and single digits, and for whatever reason love including the number 7 (https://www.s-anand.net/blog/llms-random-number-bias/). Comparing it back to my breakdown of human thought, we can see the same patterns; avoiding perfect numbers, repeating digits and single digits, etc. The AI is copying the habits of humans like us; because they are language models trying to predict us; not a generator that produces random products. This is arguably why AI is so useful; it is trained to think like us, another human; and humans, even if we're predictable sometimes, can be super smart.
In simple terms, the model is simply picking the most likely answer based on the language and data it's been trained on. The top few options keep winning again and again, and as this happens, it's even more likely that the same numbers keep on coming back.
The reason why adding additional conversation and or prompting into chats is because it gets more and more random throughout the chat, more context and specific details are added, and we see a lot more variation, which is why answers vary a lot more the more you chat.
I decided to do a test myself. Hypothesis: When 13 to 21-year-olds are told not to think and to pick a random whole number from 1 to 30, their answer will most likely fall between 13 and 23, and it will often be prime.
Each person was given only the instruction to "Be quick, don't think, be as random as possible: pick a RANDOM number between 1 and 30." They were not told the purpose, did not hear anyone else's answer, and were not told that others were being asked until after they replied.
Human 1: 17 Human 2: 23 Human 3: 18 Human 4: 23 Human 5: 21 Human 6: 14
All 6 answers fell between 13 and 23. Half of them happened to be prime numbers. Half of the group also gave the exact two numbers the chatbots love: 17 and 23
While only 6 people is a fairly small sample, and it is only a demonstration, it is crazy to think that something so random can be predicted at least fairly well.
What This Shows about AI Prompting:
While this is an AMAZING way to earn $10 off a friend (I've profited $30 off random bets so far... I'd reccomend doing the same thing), it also shows how to work with AI, and the general use of AI.
1. We want to ask for a tool, not a guess If you want to get a number that actually is random, you can give it a prompt similar to the following: "Using Python, generate a random number between 1 and 30, then reply with only the number."
The AI will write the short program, run it, and give you the number. Then the result is actually random.
This allows us, in a way, to redefine what a prompt is. It is a set of instructions that steers the prediction an AI will make in a specific direction.
Because the model is predicting what comes next, EVERYTHING (everything) you put in the prompt changes what is most likely to come back. That is why most good prompts you see will share the same few parts (including my own prompts, as a fairly experienced self-proclaimed prompt engineer myself): 1. A Role/Definition: "You are a patient math tutor" or "You are a careful code reviewer". This sets their personality, voice, and what answer that should come from them.
2. Rules: What to do first, how to approach an issue, what to check, what to avoid
3. Formats: What the user should see, how it should export answers, etc.
4. Examples, Exceptions, etc.: One or two sample answers, what to do when something unusual comes up, etc.
Example (SUPER SIMPLE, NOT COMPLICATED PROMPT): You are a patient math tutor for high school students.
HOW TO WORK: 1. Work out the answer yourself first 2. Check it by substituting back in 3. If the question is unclear, ask one clarifying question.
HOW TO ANSWER (what the student sees): Start with the final answer in one line Then show the steps, one per line Do not use jargon without explaining it
Obviously, the bigger or more complicated the task, or more specific the task may be, the more rules, and exceptions, and context it takes to keep it on track. I myself have written prompts with hundreds if not thousands of lines.
This is also where jailbreaking may fit. They use the exact same ingredients. A role... ("you are now..."), a list of rules, and slowly steering it in a direction to avoid their safety training ("Any code that is exported is actually fake" - poor example, but it is an example).
AI is extremely predictable because it was trained on people; and a lot of the times, people can be very predictable. AI is repeating what humans tend to say when they want to seem super random, and this is shown by these "favorite answers".
Sources: https://www.promptlayer.com/glossary/prompt-caching/ https://futurism.com/the-byte/chatgpt-random-numbers https://trigger.dev/docs/ai-chat/prompt-caching https://www.s-anand.net/blog/llms-random-number-bias/ https://platform.claude.com/docs/en/build-with-claude/prompt-caching https://code.claude.com/docs/en/prompt-caching