Why does prompt engineering work, since prompt engineering questions don't appear as training data?

Question

One can find advice on prompt engineering telling basically the following thing: if you are seeking advice about a topic X, start your prompt to an LLM by

You are an expert in X, you have pedagogical skills, and you are very good at synthesizing information blahblahblah...

I don't understand why this should work at all. Let's simplify the matter: assume an LLM is trained only on stackoverflow questions and answers. Since no question on SO starts with such a sentence, the pre-prompt actually makes the prompt more different from training data that it was before.

More generally, why would writing prompts like asking questions to a being, and moreover telling this being what it is give us better output? Moreover, is this even measurably true?

My hand-waving understanding of LLMs is that since they basically are excellent for predicting the following words of a context, they have, as a side-effect, learnt to answer questions just because they have learnt that on the Internet, questions are usually followed by answers.

Answer 1

LLMs are usually fine-tuned on instruction-following datasets so that they can answer user requests. For instance, ChatGPT comes from InstructGPT, which was trained from GPT-3 with reinforcement learning with human feedback (RLHF) on an instructions dataset (you can learn more about the evolution of ChatGPT/GPT-4 from the original GPT in this answer).

As for the prompts that request the LLM to impersonate a role giving better results, you can check the article In-Context Impersonation Reveals Large Language Models' Strengths and Biases, which verifies the claim. The reason for that is difficult to figure out, given the black-box nature of LLMs and deep learning in general.

Answer 2

I believe you misunderstand how LLMs work. They are not search engines that look through a data set trying to find a best match.

An LLM essentially (very simplified, and I'm not an expert) generates the most probable answer to a given question, using a generalized large data set of input data.

The "you are" and other context-giving information helps the LLM to narrow down where the most probably answer should come from. As a silly example, if I were to ask an LLM without any prompts "what is the best lunch?" it has a huge amount of data that matches "lunch" and no guidance how to rate "best", so it will mix and match everything from nutritional value to taste to ease of preparation. If I give it context information such as "as an expert on nutrition" it can weigh the nutritional value interpretation of "best" higher than, say, the taste.

On SO, context information can inform the LLM which community is more relevant to an answer, or which nearby concepts to include. While, say, "pedagogical skills" may not appear verbatim in a possible answer, the context of pedagogy is more likely to be present in a relevant community than in, say, answers about a computer game that by pure word-matching might fit.

Answer 3

There are 3 layers to the answer why:

1. ChatGPT is fine tuned with around 11,000 examples of humans asking questions, and receiving helpful and intelligent responses.
2. On the surface, LLMs have only one job. To look at as much of the preceding text as possible, and give the most likely (or one of the top few most likely) next words. By taking the lead, you can say things that would be statistically more likely to be followed by intelligent responses.
3. The transformer architecture encourages the model to look beneath the surface of language, and discover deep abstractions, and layer these abstractions into complex combinations. By triggering keywords with strong connections to your desired content, the model will be more likely to respond with the vibe you are hoping for.

And yes “vibe” is probably the most accurate way to describe this!

Answer 4

assume an LLM is trained only on stackoverflow questions and answers.

I think that it is where you are getting things wrong. LMM are trained on pile of texts (one such dataset is even known as "the pile"). Adding a prompt will help the model focus on text of better quality... but we can't really get further enough in that explanation as it is very difficult to tell exactly what is happening inside a LLM. Some of the effects are measurable (on a benchmark) but that's it.

Answer 5

I think this article is one of the best I've seen which describes LLMs in an accessible way without dumbing it down to the point where it's unhelpful: "Large language models, explained with a minimum of math and jargon"

One thing that might help you is to understand that LLMs store their 'knowledge' as vectors in a 'high-dimensional' space. As an example, here's a vector representation of 'cat':

[0.007398007903248072, 0.0029612560756504536, -0.010482859797775745, 0.0741681158542633, 0.07646718621253967, -0.0011427050922065973, 0.026497453451156616, 0.010595359839498997, 0.0190864410251379, 0.0038335588760674, -0.0468108132481575, -0.021150866523385048, 0.009098375216126442, 0.0030140099115669727, -0.05626726150512695, -0.039609555155038834, -0.09978967905044556, -0.07956799119710922, 0.057768501341342926, -0.017375102266669273, 0.015590683557093143, -0.022376490756869316, 0.10152265429496765, -0.05138462409377098, 0.025884613394737244, 0.07069036364555359, 0.0009145145886577666, -0.06275367736816406, 0.03610750287771225, 0.050807688385248184, -0.06453944742679596, -0.0434986837208271, -0.1264101266860962, -0.0003191891883034259, 0.04311852902173996, -0.14792846143245697, -0.019480768591165543, 0.01992032676935196, 0.011479354463517666, 0.02979433164000511, 0.06154156103730202, -0.04609882831573486, -0.053286727517843246, -0.016268745064735413, 0.03660176321864128, -0.07168425619602203, 0.05497466400265694, -0.1446477174758911, 0.09316877275705338, -0.1279120296239853, 0.030971739441156387, 0.03677519038319588, 0.13407474756240845, -0.028527621179819107, -0.10431249439716339, 0.03328850120306015, 0.1295083463191986, 0.0412190817296505, 0.03605308011174202, 0.0599723681807518, 0.025970442220568657, -0.03521350771188736, -0.015058198012411594, 0.005818498786538839, 0.013812823221087456, 0.015064566396176815, 0.022925062105059624, 0.039051759988069534, 0.007009583059698343, -0.02910810336470604, 0.1011449322104454, 0.13727356493473053, 0.022466043010354042, -0.07582768052816391, -0.04469817131757736, -0.06026916950941086, 0.04192522168159485, 0.1612275242805481, 0.014356226660311222, -0.0647699236869812, -0.14182332158088684, 0.07568981498479843, 0.002798931673169136, 0.012406392954289913, -0.09695082157850266, -0.0014245212078094482, -0.018527435138821602, 0.009911706671118736, 0.013058848679065704, 0.048697732388973236, 0.017661960795521736, 0.036917395889759064, 0.005680330563336611, 0.024947546422481537, 8.419259393122047e-05, -0.002204198157414794, -0.007295176852494478, 0.008355203084647655, -0.015072236768901348, -0.0032011312432587147, 0.05527794361114502, 0.020942343398928642, -0.019445667043328285, -0.15129604935646057, 0.0337672121822834, 0.0019582323729991913, -0.0014046517899259925, -0.05954226478934288, -0.08176489174365997, 0.024112699553370476, -0.1015794649720192, 0.05419696122407913, 0.13000570237636566, -0.05808615684509277, 0.004180640447884798, 0.01880498044192791, 0.01923936977982521, -0.041859131306409836, 0.010098426602780819, 0.025394367054104805, -0.03678150847554207, 0.03255629166960716, -0.008087233640253544, -0.07101460546255112, 0.024909185245633125, -0.0369131900370121, 0.035895638167858124, 0.0047763800248503685, -0.01754925213754177, -0.0029735821299254894, 0.030521586537361145, 0.04243304952979088, 0.05969628319144249, -0.07855783402919769, -0.07639002054929733, -0.004820443224161863, 0.0651308000087738, 0.13445857167243958, -0.06609761714935303, 0.01714201085269451, 0.019574925303459167, -0.00021718056814279407, 0.07559319585561752, 0.05964002385735512, -0.0715465098619461, 0.04068697988986969, -0.09640928357839584, -0.07235930114984512, -0.05935797095298767, 0.009602724574506283, -0.05649569258093834, 0.0025645969435572624, -0.05413592606782913, -0.017797887325286865, 0.05755465477705002, 0.08609342575073242, 0.050908517092466354, -0.05604008585214615, -0.005856652744114399, 0.02329830639064312, 0.08168350160121918, -0.0718611553311348, -0.027544423937797546, -0.08970167487859726, 0.024058541283011436, -0.02770240046083927, -0.025339743122458458, 0.010991393588483334, 0.02215300314128399, -0.02829679660499096, -0.07363404333591461, 0.0556303896009922, 0.0002929845068138093, -0.059732820838689804, -0.04813411086797714, -0.0021529451478272676, 0.004276854917407036, 0.04970701038837433, 0.02516869269311428, -0.05129590258002281, 0.0767771303653717, -0.08236679434776306, 0.019983036443591118, -0.05183032900094986, 0.05824366584420204, 0.047829821705818176, -0.13605566322803497, 0.02234281599521637, -0.03254450857639313, 0.011368651874363422, -0.05135396867990494, -0.00048283161595463753, -0.06719424575567245, -0.018972834572196007, 0.025254448875784874, -0.03858991339802742, 0.036364443600177765, -0.025158191099762917, 0.030907975509762764, -0.08114158362150192, 0.09369450062513351, 0.09405472874641418, 0.012534121051430702, -0.01041880901902914, 0.0552687831223011, 0.07056140154600143, 0.06628888100385666, 0.06548195332288742, 0.01580229587852955, -0.038310837000608444, -0.0032484608236700296, -0.010157674551010132, 0.085805244743824, 0.010575438849627972, 0.06210837885737419, -0.0071502267383039, -0.02955375239253044, 0.0289775263518095, 0.002539787907153368, -0.07370137423276901, 0.026873936876654625, 0.02770836278796196, 0.02373671904206276, 0.04336617887020111, 0.037974126636981964, 0.061377692967653275, 0.05020896717905998, -0.1109858900308609, -0.02423020824790001, 0.03785136342048645, 0.18769624829292297, 0.10594339668750763, -0.05118405446410179, 0.06405289471149445, -0.047474540770053864, 0.04021701216697693, -0.048911526799201965, 0.041514985263347626, -0.005742703098803759, 0.0034058222081512213, 0.01214022096246481, -0.037784647196531296, 0.008946173824369907, -0.030592333525419235, 0.039058126509189606, 0.02660788968205452, 0.05596623942255974, -0.03365514427423477, 0.09071480482816696, 0.034562114626169205, 0.08310434222221375, 0.03441822528839111, 0.003703191876411438, 0.002236866159364581, -0.06042943149805069, 0.06852643936872482, 0.09876436740159988, 0.01411499921232462, -0.07860662043094635, 0.06403335183858871, -0.1592547744512558, -0.01012679934501648, -0.10094276070594788, 0.01604175567626953, 0.006357499398291111, 0.02171235904097557, 0.01998433656990528, -0.029795801267027855, 0.020991159602999687, 0.027527112513780594, 0.07752928882837296, -0.01912834122776985, -0.10472745448350906, -0.0327356792986393, -0.11220412701368332, 0.03347017243504524, -0.04368103668093681, -0.00044717983109876513, -0.029803894460201263, 0.06123579293489456, 0.039308369159698486, -0.055449601262807846, 0.07417158037424088, -0.022331053391098976, -0.11767527461051941, -0.04385286569595337, -0.019754905253648758, 0.031432103365659714, 0.03378641605377197, 0.07572634518146515, -0.04749307036399841, -0.005324371624737978, -0.08255213499069214, -0.010222465731203556, 0.021690042689442635, -0.1339070200920105, 0.007615163456648588, -0.0929502621293068, 0.05977592244744301, 0.00015643733786419034]

The distance and mathematical relationships between vectors is an essential part of how an LLM's abilities emerge.

This is likely an over-simplificiation but the way I think about it (I may have read this) is that prompts help orient the LLM context within its vector space. Prompt engineering is essentially finding ways to shift that context to a place in that vector space where it can produce better (or worse, depending on your goal) results.

Answer 6

The short answer: engineered prompts are deliberately similar to fragments from training data, and this similarity is sufficient to elicit desired responses. Prompts do not need to literally appear in the training data.

Suppose we have a series of answers to some question, and we are training a question-answering language model. The model learns that facets of the answers are related to the question, and it also learns that facets of the answers are internally related to other facets of themselves. This latter learned behavior allows the start of an answer to causally influence the rest of that answer.

Now, suppose that some answers are written with qualifications at the start. For example, answers might start with "in my experience doing computer science," or "as a computer scientist," or "I am a computer scientist, and". These starts are similar to each other, and we can make a probabilistic generalization about their continuations: answers which start this way are likely to continue a certain way.

For the prompt engineer, this is usually enough to be able to infer that a prompted answer could start with something like, "I am a computer scientist." in order to get answers which are continued as if they were originally written with that starting prompt.

Note that this engineered prompt may not yet return decent answers. It could be the case that anybody who self-identifies as a computer scientist also writes poor answers, in which case this prompt would decrease the quality of completions. Prompt engineers have to consider word choice, just like writers in other disciplines.

Also note that part of your question is really about the usage of commands to the second-person "you". This is specific to models which have post-training (usually RLHF); they are trained to have an additional conversational behavior which doesn't necessarily exist in their training data. If I am writing on my own, then I can use a first-person perspective; I only need to talk to you in a conversational context. This is a non-trivial insight required to get conversations out of models which weren't trained for it; the model needs to be told who is "I" and "you", requiring additional prompting, framing, and parsing.