Technical Guide: Python Desktop AI Assistant on

Windows
This guide outlines how to build a Python-based desktop AI assistant on Windows that listens to voice
commands, uses an LLM (e.g. OpenAI GPT) to plan actions, controls the mouse/keyboard to execute tasks
across applications, and speaks back with a lip‐syncing 3D avatar. The system integrates several
components: a speech‐to‐text (STT) frontend, an AI interpreter, GUI automation libraries, a text‐to‐speech
(TTS) engine, and a 3D avatar renderer. Below we recommend specific libraries/APIs and describe the
architecture, file organization, and deployment steps for a fully working solution.

Recommended Tech Stack & Libraries

• Programming Language: Python 3.9+.
• Voice Input (STT): SpeechRecognition (Python) with PyAudio for microphone capture, using Google
Cloud Speech‐to‐Text or Azure Speech-to-Text as backend. SpeechRecognition supports Google’s STT
via recognizer_instance.recognize_google_cloud 1 .
• AI Instruction Parsing: OpenAI GPT via the openai Python SDK (ChatCompletion API). Optionally
use a framework like LangChain to manage prompts/chains. The GPT model (e.g. gpt-3.5-turbo or
gpt-4) processes the transcribed text and outputs structured instructions. (Example code uses
openai.ChatCompletion.create(model=..., messages=...) 2 .)
• GUI Automation (Mouse/Keyboard): PyAutoGUI for cross-platform mouse/keyboard control.
PyAutoGUI “lets your Python scripts control the mouse and keyboard to automate interactions with
other applications” 3 . It can move the mouse, click, type keystrokes, etc. For Windows‐specific
control (focusing windows by title, interacting with controls), pywinauto can also be used; e.g.
pywinauto.keyboard.send_keys() automates keystrokes to the active window 4 .
• Text-to-Speech (TTS): Azure Cognitive Services Speech SDK (Python) for high-quality neural voices.
Azure’s SpeechSynthesizer can convert text to audio (and provide viseme events if needed).
Example:

speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config,
audio_config)
result = speech_synthesizer.speak_text_async(text).get()

(This Python example is from Azure’s docs 5 .) Other options include Google Cloud TTS or third-
party APIs (Amazon Polly, ElevenLabs).
• 3D Avatar & Rendering: Ready Player Me (RPMC) to generate a realistic humanoid avatar (glTF/
GLB). RPMC provides a REST API to fetch a 3D avatar model by ID 6 . For real-time rendering and
lip-sync, use a 3D engine. Two approaches: (a) Embed a WebGL scene (Three.js or Babylon.js) in a
Python GUI (via PyWebView or QtWebEngine) to load and animate the glTF; (b) Use a Python 3D
engine like Panda3D with a glTF plugin 7 . RPMC avatars support ARKit facial blend shapes and

1
 integrate with Oculus OVR LipSync for viseme animation 8 . Example references show mapping
Azure TTS output to animate Three.js avatars 9 .
• Desktop GUI: Use a GUI toolkit (e.g. PyQt5/PySide6, Tkinter or PyWebView) to create the application
window. The GUI hosts the 3D avatar view (e.g. a QWebEngineView or PyWebView window with a
Three.js canvas) on the right side, and optionally a control panel or log on the left. The GUI thread
manages events and updates for voice commands and avatar animation.

The overall tech stack might look like:

Python 3.9+
|-- speech_recognition (with PyAudio) -> Google/Azure STT
|-- openai (ChatCompletion GPT-4/3.5)
|-- pyautogui / pywinauto for input automation
|-- azure-cognitiveservices-speech for TTS
|-- pyqt5 or pywebview for GUI
| |-- embedded WebGL (Three.js) or Panda3D for 3D avatar
|-- ReadyPlayerMe avatar (downloaded via API)

Voice Input (Speech-to-Text)

Use a microphone to capture audio and convert it to text. A common Python approach is the
SpeechRecognition library with PyAudio:

import speech_recognition as sr
r = sr.Recognizer()
with sr.Microphone() as source:
audio = r.listen(source)
try:
text = r.recognize_google_cloud(audio,
credentials_json=GOOGLE_CLOUD_CREDENTIALS_JSON)
# or r.recognize_azure(...) for Azure, etc.
except Exception as e:
print("STT error:", e)

SpeechRecognition supports various backends. For Google’s Speech-to-Text API, install the Google Cloud
Python library and use recognizer.recognize_google_cloud(...) 1 . If using Azure, the azure-
cognitiveservices-speech SDK can be used for streaming recognition. Ensure you configure the API keys/
credentials (store securely or via env variables). The transcribed text goes to the AI reasoning module.

AI Instruction Parsing (GPT)

Send the transcribed text to the OpenAI (or similar) LLM to interpret the command. For example, use the
openai Python package:

2
 import openai
openai.api_key = YOUR_API_KEY
messages = [
{"role": "system", "content": "You are an assistant that executes desktop
commands."},
{"role": "user", "content": user_text}
]
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=messages
)
assistant_reply = response.choices[0].message["content"].strip()

This code snippet (conceptually from 2 ) shows using openai.ChatCompletion.create() . The system
prompt should instruct the model to produce actionable instructions or a JSON response. For robustness,
consider wrapping the LLM call in a chain (e.g. LangChain) and/or ask it to output a structured format (like
JSON with action commands). For example, ask GPT to respond with something like {"action":
"open_app", "application": "chrome"} or plain English “opening Chrome”.

Screen Control & Automation

Once the AI has parsed the instruction, use Python automation libraries to perform the action visibly.

• PyAutoGUI: This library can move the mouse cursor, click, type text, press keys, etc., emulating a
human user. For example:

import pyautogui
pyautogui.moveTo(100, 150) # move mouse
pyautogui.click() # click
pyautogui.write('Hello!') # type text
pyautogui.press('enter') # press Enter

As the [PyAutoGUI docs] show: “PyAutoGUI lets your Python scripts control the mouse and keyboard
to automate interactions with other applications” 3 . You can script it to open menus, drag
windows, switch apps, etc.

3
Figure: PyAutoGUI automating mouse movements (the example draws a spiral in MS Paint) 3 .

• pywinauto (Windows-only): For more reliable Windows UI automation, pywinauto can send keys or
mouse events to specific windows or controls. Example:

from pywinauto.keyboard import send_keys

send_keys('%{F4}') # Alt+F4 to close the active window 4 .

pywinauto can identify windows by title and controls by name, which can complement PyAutoGUI
when needed.

Design your code so that the GPT output triggers the correct automation sequence. For instance, if GPT
returns “open Chrome and search for cats”, the code should execute pyautogui.click(...) at the Start
menu, type “Chrome”, press Enter, then wait for the browser and type a search query, etc.

Text-to-Speech (Speech Output)

Convert the assistant’s response text into spoken audio. Use Azure Cognitive Services Text-to-Speech for
high-quality neural voices (or Google Cloud TTS/other). For example, with the Azure Speech SDK:

import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(subscription=SPEECH_KEY,
region=SPEECH_REGION)
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"
audio_config = speechsdk.AudioConfig(use_default_speaker=True)
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config,

4
 audio_config=audio_config)
result = synthesizer.speak_text_async(assistant_response_text).get()

This closely follows Azure’s quickstart (above code is analogous to 5 ). The .speak_text_async() call
outputs audio to the speakers by default. You may also capture audio buffers if needed.

The chosen TTS API might also provide viseme or phoneme timing information for lip-syncing. For instance,
Azure has a VisemeReceived event in some SDKs. Alternatively, you can approximate lip movements by
feeding the generated audio (or text) into a viseme model like Oculus OVR LipSync or a custom phoneme-
to-viseme mapping.

3D Avatar Display & Lip Sync

To create a human-like animated avatar on the right side of the screen, follow these steps:

1. Avatar Model: Use Ready Player Me (RPMC) to generate a custom avatar. RPMC provides an API to
get a 3D avatar GLB/GLTF by ID 6 . You can either fetch a ready avatar or integrate the RPMC web
“Avatar Creator” to let the user pick/modify an avatar. Download the GLB file from https://
models.readyplayer.me/{avatarId}.glb .

2. Rendering Engine: Choose a real-time 3D renderer. Options include:

3. WebGL (Three.js/Babylon.js): Host a local HTML/JS scene. Many demos load GLTF avatars and
animate them via WebGL. For example, Three.js can load the RPMC GLTF and use morph targets or
bone animations for facial movement. (StackOverflow resources list demos of Azure TTS mapped to
Three.js avatars 9 and Amazon Polly with Babylon.js.)
4. Python 3D Engine: Panda3D is a Python-based 3D engine. It can load GLTF models using the
panda3d-gltf plugin ( pip install panda3d-gltf ) 7 . You can animate the model’s face by
adjusting blend shapes or bone poses in Panda3D in sync with speech.

5. Game Engine Integration: Alternatively, a Unity or Unreal component could be used (Unity has a
Ready Player Me SDK and supports Oculus OVR LipSync). If you have a Unity build, you could launch
it from Python or communicate via sockets. (The StackOverflow answer suggests loading the GLTF in
a native engine like Unity/Unreal and using their scripting 10 .)

6. Lip Sync Animation: Ready Player Me avatars include ARKit-compatible facial blend shapes
(visemes) 8 . You can map speech audio to these visemes. For example, Oculus OVR LipSync library
can take the microphone audio (or audio buffer) and output viseme blend values. When the assistant
speaks, feed the same text/audio to the LipSync engine and animate the avatar’s jaw/mouth. The
cited resources demonstrate synchronizing TTS with 3D models (e.g. Azure TTS→viseme→Three.js
animation) 9 .

7. GUI Embedding: In your Python GUI (e.g. PyQt or PyWebView window), place the 3D view on the
right side. If using a WebGL approach, embed a browser widget (QtWebEngine or PyWebView) that
loads a local HTML file running Three.js. The avatar on the right will play the lip-sync animation while
the audio plays.

5
Desktop GUI Integration
Create a desktop application window (with PyQt5/PySide6, Tkinter, or PyWebView) that organizes the
interface. A typical layout:

• Left pane: (optional) Log/controls that show recognized commands, system status, or any fallback
text.
• Right pane: The animated 3D avatar viewport (embedded webview or 3D widget).

The GUI also manages event loops. For example, in PyQt you might have a QMainWindow with a
QVBoxLayout or QHBoxLayout . Use threading or async callbacks so that the voice/STT/GPT operations
don’t freeze the UI. Update the avatar’s animation each time speech synthesis begins or a viseme event is
received. There is no specific library to cite here, but frameworks like PyQt, PySimpleGUI, or PyWebView are
commonly used to build Python GUI apps.

Sample Folder Structure

Organize the project for clarity. Example structure:

assistant_app/
├─ main.py # Entry point: initializes GUI, threads, and
services
├─ requirements.txt # pip dependencies (openai, pyautogui,
SpeechRecognition, azure-cognitiveservices-speech, PyQt5, etc.)
├─ voice/
│ ├─ stt.py # Module: microphone capture and Speech-to-Text
│ └─ tts.py # Module: Azure TTS service wrapper
├─ ai/
│ ├─ gpt_client.py # Module: calls OpenAI API, parses responses
│ └─ intent_parser.py # (Optional) turns LLM output into structured
commands
├─ automation/
│ ├─ actions.py # Module: high-level functions (e.g. open_app(),
type_text(), click_button())
│ └─ controllers.py # Uses pyautogui/pywinauto to implement the actions
├─ avatar/
│ ├─ avatar_scene.html # (If WebGL) HTML + JS to load RPMC model and
animate lip sync
│ ├─ avatar.py # (If Panda3D) Python script to load model and
update visemes
│ └─ rp_models/ # (Optional) pre-downloaded RPMC avatar files (GLB)
├─ gui/
│ ├─ interface.py # Module: creates the main GUI window and widgets
│ └─ resources/ # Icons, QML files, etc.

6
 └─ config/
└─ keys.json # (Securely) store API keys or settings

All important logic is separated by function. main.py starts the GUI and kicks off the voice-listening loop.
The automation/actions.py might contain wrappers like open_chrome() ,
type_in_notepad(text) , etc., which use PyAutoGUI internally. The avatar/ directory depends on
your avatar approach (HTML+JS files for Three.js or Panda3D Python scripts).

System Architecture & Data Flow

The system operates as follows:

1. Audio Capture: Continuously listen on the microphone. When the user speaks a command, record
the audio.
2. Speech-to-Text: Pass the audio to the STT service (e.g. Google Cloud STT or Azure STT) and receive a
text transcript.
3. AI Reasoning: Send the transcript to the LLM (OpenAI GPT). The model returns either a textual
response and/or a parsed set of actions.
4. Action Execution: Interpret the GPT output. For each detected command (e.g. “open Chrome”, “type
hello”, “scroll down”), call the appropriate automation function using PyAutoGUI/pywinauto. The
mouse moves and clicks happen in real time on the desktop, visible to the user (the assistant is
literally controlling the UI).
5. Generate Speech: The assistant’s textual reply (also from GPT) is sent to the TTS engine (Azure).
Audio is played through the speakers.
6. Avatar Animation: While the audio plays, generate lip-sync for the avatar. For example, use the
same text/audio to drive viseme animation (e.g. via Oculus OVR LipSync or a phoneme-to-viseme
mapping) so the 3D avatar’s mouth moves in sync with the voice.
7. GUI Update: The avatar on screen lip-syncs to the speech. Meanwhile, the GUI can display logs or
highlight the actions being performed (for instance, highlighting the window where a click occurred).
8. Repeat: Return to listening for the next voice command.

This loop is asynchronous: the voice input triggers both a sequence of UI actions and a spoken response by
the avatar. The cited examples of TTS-driven avatar demos 9 illustrate similar data flows. Note that if a
native engine (Unity/Unreal) were used, one could directly load the RPMC GLTF and call the TTS API from
within the engine script. However, in our Python design we treat the avatar as a separate render module
that listens for audio/viseme events from the main app.

Sample Deployment (Windows App)

To distribute the assistant as a self-contained Windows application, use PyInstaller (or similar). PyInstaller
can bundle Python scripts and dependencies into a single EXE. For example:

pyinstaller --onefile --windowed main.py

7
This will collect the Python interpreter, your code, and required libraries (SpeechRecognition, openai,
PyAutoGUI, Azure SDK, PyQt5/PyWebView, etc.). You may need to include data files (HTML, model files) via a
.spec file. Test the packaged EXE on a clean Windows machine to ensure all required DLLs (e.g. Qt or
audio codecs) are included. For engines like Panda3D, consider using the p3ddeploy or
panda3d-tools to package the application. After packaging, users can run a single executable to launch
the AI assistant.

Note: Securely handle API keys (do not hard-code them). For distribution, consider reading keys from an
encrypted local file or prompting the user.

Conclusion
By combining Python libraries for speech I/O, AI reasoning, UI automation, and 3D rendering, you can build
an AI assistant that listens, thinks, acts, and speaks in a human‐like manner. The key components are: a
reliable STT frontend (Google or Azure), an LLM backend (OpenAI), and automation tools (PyAutoGUI/
pywinauto) to make the assistant “touch” the screen. For visual feedback, a ReadyPlayerMe avatar animated
with lip-sync provides an engaging interface 11 8 . The system architecture ties them together in a real-
time loop from audio input to screen action to speech output. Finally, packaging with PyInstaller produces a
single Windows app. With this guide and the cited resources, you have a blueprint for implementation of
the full system.

Sources: We leveraged official docs and community answers for each component: Ready Player Me API
docs 6 , StackOverflow on avatar integration 11 9 10 , PyAutoGUI documentation 3 , pywinauto docs
4 , the SpeechRecognition package info 1 , Azure Speech SDK quickstart 5 , and others as cited above.

1 SpeechRecognition · PyPI
https://pypi.org/project/SpeechRecognition/

2 basic openai chat completion example · GitHub

https://gist.github.com/pszemraj/c643cfe422d3769fd13b97729cf517c5

3 Welcome to PyAutoGUI’s documentation! — PyAutoGUI documentation

https://pyautogui.readthedocs.io/en/latest/

4 pywinauto.keyboard — pywinauto 0.6.8 documentation

https://pywinauto.readthedocs.io/en/latest/code/pywinauto.keyboard.html

5 Text to speech quickstart - Speech service - Azure AI services | Microsoft Learn

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-text-to-speech

6 8 GET - 3D avatar | Ready Player Me

https://docs.readyplayer.me/ready-player-me/api-reference/rest-api/avatars/get-3d-avatars

7 glTF Files — Panda3D Manual

https://docs.panda3d.org/1.11/cpp/pipeline/gltf-files

9 10 javascript - Make a realtime realistic 3D avatar with text-to-speech, Viseme Lip-sync, and
11

emotions/gestures - Stack Overflow

https://stackoverflow.com/questions/73806104/make-a-realtime-realistic-3d-avatar-with-text-to-speech-viseme-lip-sync-and-em