LLM Production

• Lower Precision
• Flash Attention
• Architectural Innovations

분류 기준은 허깅페이스의 Optimize LLM in Production¹ 참고하여 production에 필요한 여러 구성요소를 정리해본다.

Lower Precision

Flash Attention

• flash-attention²
• Flash-Decoding for long-context inference³

• PagedAttention: vLLM

Architectural Innovations

• Improving positional embeddings of LLMs
• The key-value cache
  • Multi-Query Attention(MQA)
  • Grouped-Query Attention(GQA)

Last Modified: 2023/11/23 16:11:13