Qwen2.5 1.5B Instruct
par litert-community
Open source · 135k downloads · 39 likes
2.0
(39 avis)ChatAPI & LocalÀ propos
Le modèle Qwen2.5 1.5B Instruct est une version optimisée du modèle Qwen2.5, spécialement conçue pour fonctionner efficacement sur des appareils mobiles comme Android et iOS. Il permet d'exécuter des tâches d'inférence de langage directement sur l'appareil, sans dépendre d'un serveur externe, garantissant ainsi une meilleure confidentialité et une latence réduite. Ses principales capacités incluent la génération de texte, la compréhension de requêtes et la réponse à des questions dans un format conversationnel. Ce modèle se distingue par sa légèreté et ses performances optimisées pour les environnements mobiles, tout en offrant une bonne qualité de réponse pour des cas d'usage variés comme les assistants virtuels, les applications éducatives ou les outils de productivité.
Documentation
litert-community/Qwen2.5-1.5B-Instruct
This model provides a few variants of Qwen/Qwen2.5-1.5B-Instruct that are ready for deployment on Android using the LiteRT (fka TFLite) stack, MediaPipe LLM Inference API and LiteRT-LM.
Use the models
Colab
Disclaimer: The target deployment surface for the LiteRT models is Android/iOS/Web and the stack has been optimized for performance on these targets. Trying out the system in Colab is an easier way to familiarize yourself with the LiteRT stack, with the caveat that the performance (memory and latency) on Colab could be much worse than on a local device.
Android
Edge Gallery App
-
Download or build the app from GitHub.
-
Install the app from Google Play.
-
Follow the instructions in the app.
LLM Inference API
- Download and install the apk.
- Follow the instructions in the app.
To build the demo app from source, please follow the instructions from the GitHub repository.
iOS
- Clone the MediaPipe samples repository and follow the instructions to build the LLM Inference iOS Sample App using XCode.
- Run the app via the iOS simulator or deploy to an iOS device.
Performance
Android
Note that all benchmark stats are from a Samsung S25 Ultra and multiple prefill signatures enabled.
| Backend | Quantization scheme | Context length | Prefill (tokens/sec) | Decode (tokens/sec) | Time-to-first-token (sec) | Model size (MB) | Peak RSS Memory (MB) | GPU Memory (RSS in MB) | |
|---|---|---|---|---|---|---|---|---|---|
CPU | fp32 (baseline) | 1280 | 49.50 | 10 tk/s | 21.25 s | 6182 MB | 6254 MB | N/A | |
CPU | dynamic_int8 | 1280 | 297.58 | 34.25 tk/s | 3.71 s | 1598 MB | 1997 MB | N/A | |
CPU | dynamic_int8 | 4096 | 162.72 tk/s | 26.06 tk/s | 6.57 s | 1598 MB | 2216 MB | N/A | |
GPU | dynamic_int8 | 1280 | 1667.75 tk/s | 30.88 tk/s | 3.63 s | 1598 MB | 1846 MB | 1505 MB | |
GPU | dynamic_int8 | 4096 | 933.45 tk/s | 27.30 tk/s | 4.77 s | 1598 MB | 1869 MB | 1505 MB |
- For the list of supported quantization schemes see supported-schemes. For these models, we are using prefill signature lengths of 32, 128, 512 and 1280.
- Model Size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models)
- Memory: indicator of peak RAM usage
- The inference on CPU is accelerated via the LiteRT XNNPACK delegate with 4 threads
- Benchmark is run with cache enabled and initialized. During the first run, the time to first token may differ.
Liens & Ressources