Summary
Highlights
The video introduces "Pocket," a small yet capable AI assistant running entirely on a Raspberry Pi. The creator highlights the benefits of local AI, including being 100% free, private, and secure, as user data remains local. However, the limitation of running large models on a Raspberry Pi 5 is acknowledged, leading to the decision to use a network of smaller models for efficiency.
The creator discusses selecting a main reasoning model, testing Quen, Deepseek, and Gemma for speed, memory usage, and intelligence. Quen is chosen due to its speed and the 'thinking model' capability. To optimize performance, a routing system is implemented, directing simple prompts to a non-thinking version of Quen for speed, and complex prompts to the thinking version for accuracy.
To enable voice interaction, speech-to-text (STT) and text-to-speech (TTS) functionalities are added. Fast Whisper is selected for STT due to its accuracy, despite a slight delay. For TTS, Piper TTS is used, with the medium-sized model chosen for its balance of quality and speed.
To allow Pocket to perform actions like fetching weather or scanning networks, a separate model, Function Gemma, is integrated for tool calling. Function Gemma, a tiny model specifically trained for outputting actions, required fine-tuning with a custom dataset to accurately call functions. The fine-tuned model successfully performs various tasks.
The Raspberry Pi is made portable by adding a battery power supply. Vision capabilities are introduced with a 12-megapixel Arducam and a Halo 8 hat for running computer vision models directly on the device without burdening the main AI. A 4.3-inch touchscreen is also integrated for user interaction, though the creator notes a 5-inch screen with built-in speakers and mic would be ideal.
A 3D-printed case is designed with ample ventilation to manage heat generated by the components. The video concludes with a demonstration of Pocket's features, including conversational interaction, tool execution (like getting stock prices or scanning networks), task scheduling for automated actions, and object detection using the integrated camera and Halo hat.