FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices.

arXiv:2501.07139v2 Announce Type: replace Abstract: Deploying LLMs on edge devices presents serious technical challenges. Memory elasticity is crucial for edge devices with unified memory, where memory is shared and fluctuates dynamically. Existing solutions s