Stealing Part of a Production Language Model

The first practical model-extraction attack against deployed black-box LLMs: by querying the production API, the authors recover the embedding-projection (final-layer) weights and hidden dimension of OpenAI and Google PaLM models for a few hundred to a few thousand dollars. A landmark security result — ICML 2024 Best Paper.

From Florian Tramèr's SPY Lab (joint→Google DeepMind, with Carlini et al.; Tramèr and Paleka on the ETH side). Prompted API providers to change how logit/logprob endpoints are exposed.