PICS Colloquium with David Schwab: “Out-of-distribution generalization in context”
Penn Institute for Computational Science 3401 Walnut Street, 5th Floor, PhiladelphiaAbstract: In-context learning (ICL) is an emergent capability of pretrained transformers that allows models to generalize to previously unseen tasks after seeing only a few examples. We investigate empirically the conditions necessary on the pretraining distribution for ICL to emerge and generalize out-of-distribution. We find that as task diversity increases, transformers undergo a transition from […]
