The fact that they “average” their inputs is why there are comparatively few typos (different sources have different typos, so they average out), not too many emojis or internet lingo (again, different sources use different ones in different places, so they average away), and why they produce such tedious stock output (it’s an average of the inputs, so all the little quirks and idioms that make human communucation more vibrant have been blended away).
I’m sure there is some filtering on the inputs to try to remove the worst of it, but ultimately it’s still just taking the rest and building it’s probability tables from that, which leads to the homogenised outputs we see.
Mind you, having said there are fewer typos, the last time I bothered trying to get one to write some code, it managed to misspell a popular library name in multiple places, which gives some indication of how bad the inputs are, how bad the tokeniser is, or possibly both.
If “different typos” averaged out to “nearly no typos” the same logic would have different words average out to nearly no words. What actually happens is the model learns context, and can produce output which contains emojis in one context and not others. These contexts can be very far from the average context.
I’m afraid the upshot is you don’t understand how the models work. There is extensive filtering before training - they do not get “the entire internet” and average it. If you want to understand properly, there are a lot of resources that will let you, but I’m not going to try to do it here, so you’ll either have to believe me or be wrong, I’m afraid.
The fact that they “average” their inputs is why there are comparatively few typos (different sources have different typos, so they average out), not too many emojis or internet lingo (again, different sources use different ones in different places, so they average away), and why they produce such tedious stock output (it’s an average of the inputs, so all the little quirks and idioms that make human communucation more vibrant have been blended away).
I’m sure there is some filtering on the inputs to try to remove the worst of it, but ultimately it’s still just taking the rest and building it’s probability tables from that, which leads to the homogenised outputs we see.
Mind you, having said there are fewer typos, the last time I bothered trying to get one to write some code, it managed to misspell a popular library name in multiple places, which gives some indication of how bad the inputs are, how bad the tokeniser is, or possibly both.
If “different typos” averaged out to “nearly no typos” the same logic would have different words average out to nearly no words. What actually happens is the model learns context, and can produce output which contains emojis in one context and not others. These contexts can be very far from the average context.
I’m afraid the upshot is you don’t understand how the models work. There is extensive filtering before training - they do not get “the entire internet” and average it. If you want to understand properly, there are a lot of resources that will let you, but I’m not going to try to do it here, so you’ll either have to believe me or be wrong, I’m afraid.