Optimize macos frame rate 14631050807658318954#1617
Open
itsgiddd wants to merge 4 commits intohacksider:mainfrom
Open
Optimize macos frame rate 14631050807658318954#1617itsgiddd wants to merge 4 commits intohacksider:mainfrom
itsgiddd wants to merge 4 commits intohacksider:mainfrom
Conversation
- Modify `modules/face_analyser.py` to use a smaller detection size `(320, 320)` on macOS for faster processing.
- Modify `modules/processors/frame/face_swapper.py` to:
- Increase `DETECTION_INTERVAL` to 0.08s for Apple Silicon to reduce detection frequency.
- Skip opacity blending if opacity is 1.0.
- Use `get_faces_optimized` in `process_frame_v2` for live streams to enable caching.
- Ensure `DETECTION_INTERVAL` is conditional based on platform.
- modules/face_analyser.py:
- Reduce face detection size to (320, 320) on macOS to improve performance.
- modules/processors/frame/face_swapper.py:
- Increase `DETECTION_INTERVAL` to 0.08s for Apple Silicon.
- Implement adaptive detection rate using `get_faces_optimized`.
- Skip opacity blending if opacity is 1.0.
- modules/ui.py:
- Remove redundant image resizing logic and expensive `ImageOps.contain` call in the live preview loop.
- Ensure consistent frame resizing before processing.
- Add `install-mac.sh`: Automated setup script for macOS (Apple Silicon). - Installs Python 3.10 via Homebrew. - Creates virtual environment. - Installs dependencies from `requirements.txt`. - Ensures proper `onnxruntime-silicon` configuration. - Add `run-mac.sh`: Optimized launcher for macOS. - Activates venv. - Runs `run.py` with `--execution-provider coreml` and optimal memory settings. - Optimizations (Confirmed): - `modules/face_analyser.py`: Reduced detection size (320x320) for macOS. - `modules/processors/frame/face_swapper.py`: Adaptive face detection rate (0.08s interval) for live streams. - `modules/ui.py`: Removed redundant image resampling in preview loop.
Contributor
Reviewer's GuideOptimizes live webcam processing and face swapping performance on macOS (especially Apple Silicon) by simplifying preview resizing, reducing redundant image scaling, adapting detection frequency and detector resolution for macOS, switching to an optimized face detection path, and adding dedicated macOS install/run scripts. Sequence diagram for optimized live webcam frame processing on macOSsequenceDiagram
actor User
participant Camera
participant UI as ui_create_webcam_preview
participant Swapper as frame_face_swapper_process_frame_v2
participant Detector as get_faces_optimized
participant Analyser as face_analyser_get_face_analyser
User->>Camera: Enable_live_webcam
Camera-->>UI: temp_frame
UI->>UI: Optional_mirror_flip
UI->>UI: fit_image_to_size(temp_frame, PREVIEW_width, PREVIEW_height)
UI->>Swapper: process_frame_v2(temp_frame)
activate Swapper
Swapper->>Detector: get_faces_optimized(processed_frame)
activate Detector
Detector->>Analyser: get_face_analyser()
activate Analyser
Analyser-->>Detector: FACE_ANALYSER_with_det_size(320x320_on_macos)
deactivate Analyser
Detector-->>Swapper: detected_faces
deactivate Detector
Swapper->>Swapper: swap_face_for_each_detected_face
Swapper->>Swapper: Apply_opacity_blend_if_opacity_lt_1_0
Swapper-->>UI: final_swapped_frame
deactivate Swapper
UI->>UI: cvtColor_BGR_to_RGB
UI->>UI: Create_PIL_Image
UI->>UI: Create_CTkImage_without_additional_contain
UI->>UI: preview_label.configure(image)
UI->>UI: ROOT.update()
UI-->>User: Updated_live_preview_frame
Class diagram for updated frame processing and face analysis modulesclassDiagram
class ui_module {
+create_webcam_preview(camera_index int)
-PREVIEW
-ROOT
}
class frame_face_swapper_module {
+FRAME_CACHE
+FACE_DETECTION_CACHE
+LAST_DETECTION_TIME
+DETECTION_INTERVAL
+FRAME_SKIP_COUNTER
+ADAPTIVE_QUALITY
+process_frame_v2(temp_frame Frame, temp_frame_path str) Frame
+swap_face(source_face Face, target_face Face, temp_frame Frame) Frame
}
class face_analyser_module {
+FACE_ANALYSER
+get_face_analyser() Any
}
class Face {
}
class Frame {
}
ui_module --> frame_face_swapper_module : uses_process_frame_v2
frame_face_swapper_module --> face_analyser_module : uses_get_face_analyser
frame_face_swapper_module --> Face : operates_on
frame_face_swapper_module --> Frame : processes
class DetectionIntervalBehavior {
+IS_APPLE_SILICON
+DETECTION_INTERVAL_apple_silicon float 0_08
+DETECTION_INTERVAL_default float 0_033
}
frame_face_swapper_module --> DetectionIntervalBehavior : adaptive_detection_interval
class MacOSFaceDetectorConfig {
+platform_system() str
+det_size_default tuple 640_640
+det_size_macos tuple 320_320
}
face_analyser_module --> MacOSFaceDetectorConfig : adaptive_det_size
class OpacityBlendLogic {
+opacity float
+blend_if_opacity_lt_1_0(original_frame Frame, swapped_frame Frame) Frame
+use_swapped_frame_if_opacity_eq_1_0(swapped_frame Frame) Frame
}
frame_face_swapper_module --> OpacityBlendLogic : controls_final_swapped_frame
Flow diagram for macOS-specific face detection optimizationsflowchart LR
A["Incoming_frame_from_webcam"] --> B["ui_create_webcam_preview<br/>Mirror_optional"]
B --> C["fit_image_to_size_to_preview_dimensions"]
C --> D["frame_face_swapper_process_frame_v2"]
subgraph Detection_Interval
D --> E{IS_APPLE_SILICON}
E -->|Yes| F["DETECTION_INTERVAL_0_08_seconds"]
E -->|No| G["DETECTION_INTERVAL_0_033_seconds"]
F --> H["Adaptive_face_detection_timing"]
G --> H
end
D --> I["get_faces_optimized"]
I --> J["fac
1000
e_analyser_get_face_analyser"]
J --> K{platform_system_is_darwin}
K -->|Yes| L["det_size_320x320"]
K -->|No| M["det_size_640x640"]
L --> N["FaceAnalysis_prepare_with_small_input_on_macos"]
M --> N
N --> O["Detected_faces"]
O --> P["swap_face_with_optional_opacity_blend"]
P --> Q["Final_swapped_frame_returned_to_ui"]
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
face_analyser.py, you now useplatform.system()to setdet_size, but there’s noimport platformin the diff; double‑check that the module importsplatformat the top to avoid a NameError. - The adaptive
DETECTION_INTERVALbased only onIS_APPLE_SILICONis now hardcoded; consider making this value configurable (e.g., via a setting or environment variable) so users can tune detection frequency for different performance/quality trade-offs.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `face_analyser.py`, you now use `platform.system()` to set `det_size`, but there’s no `import platform` in the diff; double‑check that the module imports `platform` at the top to avoid a NameError.
- The adaptive `DETECTION_INTERVAL` based only on `IS_APPLE_SILICON` is now hardcoded; consider making this value configurable (e.g., via a setting or environment variable) so users can tune detection frequency for different performance/quality trade-offs.
## Individual Comments
### Comment 1
<location> `modules/ui.py:1022-1025` </location>
<code_context>
- image = ImageOps.contain(
- image, (temp_frame.shape[1], temp_frame.shape[0]), Image.LANCZOS
- )
+ # Skip ImageOps.contain as frame is already resized
image = ctk.CTkImage(image, size=image.size)
preview_label.configure(image=image)
ROOT.update()
</code_context>
<issue_to_address>
**suggestion (performance):** Avoid redundant size argument when constructing CTkImage
Because `temp_frame` is already resized to the preview dimensions and `image` comes directly from that frame, `image.size` is already the desired size. Passing `size=image.size` to `CTkImage` is redundant and still runs the resize path; you can omit `size` and let `CTkImage` use the native size to avoid the extra no-op scaling.
```suggestion
image = Image.fromarray(image)
# temp_frame is already resized to preview dimensions; use native image size
image = ctk.CTkImage(image)
preview_label.configure(image=image)
```
</issue_to_address>
### Comment 2
<location> `install-mac.sh:16-19` </location>
<code_context>
+fi
+
+# Check for Python 3.10
+if ! command -v python3.10 &> /dev/null; then
+ echo "⚠️ Python 3.10 not found. Installing via Homebrew..."
+ brew install python@3.10
+ brew install python-tk@3.10
+else
+ echo "✅ Python 3.10 found."
</code_context>
<issue_to_address>
**issue:** Be cautious about relying on a specific Homebrew formula name for python-tk
Hard-coding `python-tk@3.10` assumes this exact formula exists and stays stable. On some systems Tk may be bundled with `python@3.10` or exposed under a different name, causing `brew install` to fail and the script to exit due to `set -e`. Consider making this install best-effort (e.g., ignore or log failures) or first detect whether the formula exists before installing it.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
1022
to
1025
| image = Image.fromarray(image) | ||
| image = ImageOps.contain( | ||
| image, (temp_frame.shape[1], temp_frame.shape[0]), Image.LANCZOS | ||
| ) | ||
| # Skip ImageOps.contain as frame is already resized | ||
| image = ctk.CTkImage(image, size=image.size) | ||
| preview_label.configure(image=image) |
Contributor
There was a problem hiding this comment.
suggestion (performance): Avoid redundant size argument when constructing CTkImage
Because temp_frame is already resized to the preview dimensions and image comes directly from that frame, image.size is already the desired size. Passing size=image.size to CTkImage is redundant and still runs the resize path; you can omit size and let CTkImage use the native size to avoid the extra no-op scaling.
Suggested change
| image = Image.fromarray(image) | |
| image = ImageOps.contain( | |
| image, (temp_frame.shape[1], temp_frame.shape[0]), Image.LANCZOS | |
| ) | |
| # Skip ImageOps.contain as frame is already resized | |
| image = ctk.CTkImage(image, size=image.size) | |
| preview_label.configure(image=image) | |
| image = Image.fromarray(image) | |
| # temp_frame is already resized to preview dimensions; use native image size | |
| image = ctk.CTkImage(image) | |
| preview_label.configure(image=image) |
Comment on lines
+16
to
+19
| if ! command -v python3.10 &> /dev/null; then | ||
| echo "⚠️ Python 3.10 not found. Installing via Homebrew..." | ||
| brew install python@3.10 | ||
| brew install python-tk@3.10 |
Contributor
There was a problem hiding this comment.
issue: Be cautious about relying on a specific Homebrew formula name for python-tk
Hard-coding python-tk@3.10 assumes this exact formula exists and stays stable. On some systems Tk may be bundled with python@3.10 or exposed under a different name, causing brew install to fail and the script to exit due to set -e. Consider making this install best-effort (e.g., ignore or log failures) or first detect whether the formula exists before installing it.
- modules/face_analyser.py: Add missing `import platform`.
- modules/processors/frame/face_swapper.py:
- Make `DETECTION_INTERVAL` configurable via `DLC_DETECTION_INTERVAL` environment variable.
- Default to 0.08s for Apple Silicon and 0.033s for others if not set.
- (Preserves previous optimizations including `get_faces_optimized` and opacity check).
laurigates
added a commit
to laurigates/Deep-Live-Cam
that referenced
this pull request
Feb 22, 2026
Switch _build_pairs_live() from get_many_faces() to get_faces_optimized() so map-faces live mode benefits from time-based detection caching on Apple Silicon, matching the simple-mode live path. Inspired by hacksider#1617. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Relevant finding from testing the InsightFace This means if the face analyzer is first initialized with Benchmark Data (M4 Pro, CoreML)
The 160x160 → 320x320 switch between live mode and video processing gives the best trade-off: live mode gets ~4x fewer detection FLOPs for smooth preview, while video/image processing retains full accuracy. Just flagging the singleton behavior since it's easy to miss and can make the optimization silently ineffective. |
laurigates
added a commit
to laurigates/Deep-Live-Cam
that referenced
this pull request
Feb 22, 2026
Switch process_frame_v2() live path from get_many_faces() to get_faces_optimized() so map-faces live mode benefits from time-based detection caching, matching the simple-mode live path. get_faces_optimized() returns cached results when called within the detection interval (~33ms), avoiding redundant face detection on consecutive frames. This was already used in the simple (non-map) live mode but the map-faces live branch was still calling get_many_faces() directly. Inspired by hacksider#1617. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 tasks
laurigates
added a commit
to laurigates/Deep-Live-Cam
that referenced
this pull request
Feb 23, 2026
Switch _build_pairs_live() from get_many_faces() to get_faces_optimized() so map-faces live mode benefits from time-based detection caching on Apple Silicon, matching the simple-mode live path. Inspired by hacksider#1617. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Improve live face swap performance and macOS support, particularly on Apple Silicon, while keeping visual behavior consistent.
New Features:
Bug Fixes:
Enhancements:
Build:
Deployment: