Skull-GPT

This repo contains software, firmware and detailed instructions that allow anyone create talking toys using LLMs and speech synthesis models. The code is optimised for real-time streaming with sub-second latency, interruptions and voice-activations.

Demos

1. https://x.com/lizziepika/status/1719190574337765437?s=20

Getting Started (generated by gpt, be carefull)

The very silly version requires:

• Arduino with Serial interface
• Python 3.x with dependencies:

  pip install -r requirements.txt

• .env file with OPENAI_API_KEY, ELEVENLABS_API and PICO_ACCES_KEY (optional)
• one skull with a motorized jaw and LED eyes.

Functionality

This script connects to an Arduino via serial, simulates an "evil bro skull" chatbot for Halloween using GPT-3.5-turbo and ElevenLabs API for voice generation. It reads user inputs, processes them, and plays the voice back.

How to Run

1. Load the required Arduino code onto your Arduino.
2. Connect the Arduino to your computer.
3. Fill in your .env file with the API keys.
4. To chat with skull, run:

   python skull-gpt-chat-stream.py

5. To speak with skull, run:

   python skull-gpt-voice.py

Latencies breakdown on a good wifi

• speech recognition: 1.5s (low hand fruit, contribute to *-hotword.py and fir mic issue)
• from prompt text responce generation start: ~0.5s
• from prompt to voice generation start: ~1.5s

Known Issues

• skull-gpt-hotword.py gives ||PaMacCore (AUHAL)|| Error on line 2523: err='-50', msg=Unknown Error
• pls help fix i'll buy u juice

TODO

• Change Arduino Nano to wifi controller with Mic to remove usb cord
• Fix the hotword detection (Error on line 2523: err='-50') (partially fixed by using skull-gpt-hotword.py)
• Optimise SpeechRecognition library to reduce latency, like use simpler model or tune pause threshold
• Split ElevenLabs streaming and Arduino skull movements into two threads to syncronise jaw movements and speech.
• Add agency features to the skull and function calling to change ElevenLabs voice, draw spooky Dalle pictures, make prank calls with Twilio, etc.
• Collect OpanAI responses in batches concurently with ElevenLabs streaming to reduce latency for large skull responses.

Acknowledgements

Thank you Noisebridge for using some of your parts. Thank you GPT for 101 on simple driver circuits and microcontrollers.

Further Notes

Try adapting the hack to breathe life to other objects and ping me x.com/savakholin for any questions. Have fun.