The headlines about models like GPT mention "parameters" a lot. I thought I knew roughly what that meant. I asked GPT chat and it (I'm just gonna, like, paste it verbatim) said...

... Imagine you're baking a cake, and you have a recipe that tells you how much of each ingredient to use. For example, the recipe might say "use 2 cups of flour, 1 cup of sugar, and 1/2 cup of butter."

Now, imagine you have a really, really complicated recipe for a cake that has a ton of ingredients. Instead of just a few ingredients, it might have hundreds or even thousands. And, instead of just one recipe, you have a whole bunch of different recipes, all mixed together.

The "parameters" in a language model like GPT are kind of like the ingredients in that cake recipe. Just like you need to use the right amount of each ingredient to make a good cake, the model needs the right "amount" of each parameter to work well.

In the case of GPT, each parameter is a number that the model uses to figure out how to generate text. The more parameters the model has, the more complex the recipe is, and the better it can do at generating text.

I think that's what I thought. Pretty much.

I wonder, even if it's "just a theory" that the human mind works kind of like this, can one take a black-box network and try to quantify its parameters? Can we use that as a quantity that could sort of indicate "smartness" in an entity. Could the IMF record this value for each of its human minions deep in a limestone cave in Austria? Probably.

So we could say, "Johnny over here is 47 quintillion parameters smart!"

Of course we also need to consider Johnny's I/O capabilities as well as his clock speed.

It is kind of a measure of: "How deeply the system can engage with the data, should it come along."