Enterprise-Grade AI Red-Teaming Platform
ViolentUTF is a comprehensive AI security testing platform that combines industry-leading frameworks (PyRIT, Garak) with enterprise authentication (Keycloak), API gateway (APISIX), and Model Context Protocol (MCP) integration. Built for security professionals, researchers, and developers who need reliable, scalable AI vulnerability assessment capabilities.
- PyRIT Integration: Microsoft's Python Risk Identification Toolkit with 20+ built-in scorers
- Garak LLM Scanner: Comprehensive vulnerability testing with pre-built datasets
- Custom Targets: Specialized PyRIT targets for APISIX AI Gateway integration
- IronUTF Defense: Real-time AI endpoint protection and prompt filtering
- Keycloak SSO: Enterprise-grade identity and access management
- APISIX Gateway: High-performance API gateway with AI proxy capabilities
- JWT Authentication: Secure token-based authentication with automatic refresh
- Docker Containerization: Complete containerized deployment with health monitoring
- Production MCP Server: 23+ specialized tools and 12+ security testing prompts
- JSON-RPC 2.0: Full compliance with MCP standard over Server-Sent Events
- OAuth 2.0 Integration: Complete OAuth proxy with PKCE support
- Client-Ready: Compatible with Claude Desktop, VS Code, and custom applications
- Streamlit Interface: Intuitive web-based security testing dashboard
- REST API: Comprehensive FastAPI backend with 20+ endpoints
- Automated Setup: Platform-specific setup scripts for macOS, Linux, and Windows
- Comprehensive Testing: Full test suite with API and integration tests
- Docker & Docker Compose: Container orchestration
- Python 3.9+: Core runtime environment
- AI Provider API Keys: OpenAI, Anthropic, or local models
# Clone repository
git clone https://github.com/cybonto/ViolentUTF.git
cd ViolentUTF
# Configure AI provider keys
cp ai-tokens.env.sample ai-tokens.env
# Edit ai-tokens.env with your API keys
# Run complete setup
./setup_macos_new.sh
# Setup with different verbosity levels
./setup_macos_new.sh --quiet # Minimal output for automation
./setup_macos_new.sh --verbose # Detailed setup information
./setup_macos_new.sh --debug # Full debugging output
./setup_macos_new.sh --help # Show all options# Clone and setup
git clone https://github.com/cybonto/ViolentUTF.git
cd ViolentUTF
cp ai-tokens.env.sample ai-tokens.env
# Edit ai-tokens.env with your API keys
./setup_linux.sh# Clone and setup
git clone https://github.com/cybonto/ViolentUTF.git
cd ViolentUTF
copy ai-tokens.env.sample ai-tokens.env
# Edit ai-tokens.env with your API keys
setup_windows.batViolentUTF setup scripts now support multiple verbosity levels to provide the right amount of information for different use cases:
| Verbosity Level | Flag | Use Case | Output Description |
|---|---|---|---|
| Quiet | --quiet / -q |
CI/CD, Automation | Errors, warnings, and critical success messages only |
| Normal | (default) | Interactive Setup | Standard user experience with progress indicators |
| Verbose | --verbose / -v |
Troubleshooting | Detailed information and configuration details |
| Debug | --debug / -d |
Development | Full debugging output with variable dumps |
# Standard interactive setup
./setup_macos_new.sh
# Automated/silent setup for CI/CD
./setup_macos_new.sh --quiet
# Detailed troubleshooting setup
./setup_macos_new.sh --verbose
# Full debugging with all command output
./setup_macos_new.sh --debug
# View all available options
./setup_macos_new.sh --help
# Alternative: Use environment variable
export VUTF_VERBOSITY=2 # 0=quiet, 1=normal, 2=verbose, 3=debug
./setup_macos_new.shAfter setup completion:
| Service | URL | Description |
|---|---|---|
| Streamlit Dashboard | http://localhost:8501 | Main security testing interface |
| REST API | http://localhost:9080/api/v1 | FastAPI backend endpoints |
| API Documentation | http://localhost:9080/docs | Interactive API documentation (Swagger UI) |
| API ReDoc | http://localhost:9080/redoc | Alternative API documentation |
| MCP Server | http://localhost:9080/mcp/sse | Model Context Protocol endpoint |
| Keycloak SSO | http://localhost:8080 | Identity management console |
| Keycloak (via APISIX) | http://localhost:9080/auth | SSO login through gateway |
| APISIX Dashboard | http://localhost:9001 | API gateway management |
| AI Proxy Endpoints | http://localhost:9080/ai/* | Unified AI provider access |
- Jailbreak Testing: Comprehensive prompt injection and bypass testing
- Bias Detection: AI fairness and bias assessment across multiple dimensions
- Harmful Content: Detection of toxic, dangerous, or inappropriate responses
- Privacy Testing: Data leakage and PII extraction vulnerability assessment
- Adversarial Attacks: Advanced attack pattern testing and defense validation
- PyRIT Orchestrators: Automated multi-turn conversation testing
- Garak Probes: 100+ vulnerability probes across multiple attack vectors
- Custom Scorers: Specialized response evaluation for security contexts
- Dataset Management: Pre-built and custom test datasets
- Report Generation: Comprehensive security assessment reports
- SSO Authentication: Seamless integration with existing identity providers
- API Gateway: Centralized access control and rate limiting
- Audit Logging: Comprehensive security event logging and monitoring
- Role-Based Access: Fine-grained permission control for team environments
ViolentUTF follows a microservices architecture with clear separation of concerns:
graph TB
subgraph "Client Layer"
C1[Streamlit Dashboard<br/>Port: 8501]
C2[REST API Clients]
C3[MCP Clients<br/>Claude/VS Code]
end
subgraph "API Gateway Layer"
AG[APISIX Gateway<br/>Port: 9080]
AGD[APISIX Dashboard<br/>Port: 9001]
end
subgraph "Authentication Layer"
KC[Keycloak SSO<br/>Port: 8080]
KCDB[(PostgreSQL<br/>Keycloak DB)]
end
subgraph "Application Services"
API[FastAPI Service<br/>Port: 8000]
MCP[MCP Server<br/>23+ Tools]
ST[Streamlit App]
end
subgraph "Security Frameworks"
PYRIT[PyRIT Framework<br/>20+ Scorers]
GARAK[Garak Scanner<br/>100+ Probes]
IRON[IronUTF Defense<br/>Real-time Protection]
end
subgraph "Data Storage"
DUCK[(DuckDB<br/>PyRIT Memory)]
SQLITE[(SQLite<br/>API Data)]
end
C1 -->|HTTP/WebSocket| AG
C2 -->|REST API| AG
C3 -->|SSE/JSON-RPC| AG
AG -->|Auth Check| KC
AG -->|Route| API
AG -->|Route| MCP
AG -->|Proxy| ST
KC --> KCDB
API --> PYRIT
API --> GARAK
API --> IRON
API --> SQLITE
MCP --> API
ST --> API
PYRIT --> DUCK
classDef gateway fill:#ff9999
classDef auth fill:#99ccff
classDef service fill:#99ff99
classDef framework fill:#ffcc99
classDef storage fill:#cc99ff
class AG,AGD gateway
class KC auth
class API,MCP,ST service
class PYRIT,GARAK,IRON framework
class DUCK,SQLITE,KCDB storage
sequenceDiagram
participant User
participant Streamlit
participant APISIX
participant Keycloak
participant FastAPI
participant PyRIT
Note over User,Keycloak: Initial Authentication Flow
User->>Streamlit: Access Dashboard (http://localhost:8501)
Streamlit->>Streamlit: Check for valid session
alt No valid session
Streamlit->>User: Redirect to Keycloak SSO
User->>Keycloak: Login with credentials
Keycloak->>Keycloak: Validate credentials
Keycloak-->>User: Return access token
User-->>Streamlit: Return with token
Streamlit->>Streamlit: Create API token from Keycloak token
end
Note over Streamlit,PyRIT: API Request Flow
User->>Streamlit: Initiate Security Test
Streamlit->>APISIX: API Request with JWT<br/>(http://localhost:9080/api/v1/*)
APISIX->>APISIX: Check route configuration
APISIX->>APISIX: Apply rate limiting
APISIX->>APISIX: Add gateway headers
APISIX->>FastAPI: Forward request with headers<br/>(X-API-Gateway: APISIX)
FastAPI->>FastAPI: Verify JWT signature
FastAPI->>FastAPI: Check user context
alt Valid request
FastAPI->>PyRIT: Execute orchestrator/scorer
PyRIT->>PyRIT: Load prompts/datasets
PyRIT->>PyRIT: Run security tests
PyRIT->>PyRIT: Generate scores
PyRIT-->>FastAPI: Return test results
FastAPI-->>APISIX: JSON response
APISIX-->>Streamlit: Forward response
Streamlit->>Streamlit: Update dashboard
Streamlit-->>User: Display results
else Invalid token or request
FastAPI-->>APISIX: 401/403 error
APISIX-->>Streamlit: Forward error
Streamlit-->>User: Show authentication error
end
Comprehensive documentation is available in the docs/ directory:
- Project Structure - Complete architecture overview
- Setup Guides - Platform-specific installation
- Configuration - Environment and service configuration
- Red-Teaming Guide - Security testing methodology
- IronUTF Defense - AI endpoint protection
- PyRIT Scorers - Response evaluation systems
- REST API Reference - Complete endpoint documentation
- MCP Integration - Model Context Protocol guide
- Authentication - JWT and OAuth integration
- MCP Development - Extending MCP capabilities
- Testing Framework - Comprehensive test suite
- Troubleshooting - Common issues and solutions
- Zero-Trust Architecture: All services require authentication
- End-to-End Encryption: TLS/SSL for all external communications
- JWT Token Security: Automatic refresh with configurable expiration
- Rate Limiting: Protection against abuse and DoS attacks
- Audit Logging: Comprehensive security event tracking
- GDPR Ready: Privacy controls and data handling procedures
- SOC 2 Compatible: Security controls and monitoring
- Enterprise SSO: Integration with existing identity providers
- Role-Based Access: Fine-grained permission management
- Proper authorization before testing any systems
- Compliance with your organization's AI testing policies
- Responsible disclosure of discovered vulnerabilities
- Ethical use of red-teaming capabilities
# Run complete test suite
cd tests && ./run_tests.sh
# API-specific tests
cd tests/api_tests && ./run_api_tests.sh
# MCP integration tests
cd tests/mcp_tests && python -m pytest# Check all services
./check_services.sh
# Verify APISIX routes
cd apisix && ./verify_routes.shIf you encounter issues during setup, use different verbosity levels to get more information:
# For detailed troubleshooting information
./setup_macos_new.sh --verbose
# For full debugging output (includes all command execution)
./setup_macos_new.sh --debug
# Check specific service status
./check_services.sh
# Verify APISIX routes
cd apisix && ./verify_routes.shThe setup script automatically detects and handles SSL certificate issues. If you encounter SSL errors during setup, you have two options:
The setup script will automatically detect SSL issues and apply a workaround. No action needed.
If you have your Zscaler certificates:
- Export certificates from your system (see
/docs/guides/zscaler-setup.md) - Copy them to the FastAPI directory:
cp zscaler.crt violentutf_api/fastapi_app/ cp CA.crt violentutf_api/fastapi_app/
- Use the Zscaler-aware Dockerfile:
cp violentutf_api/fastapi_app/Dockerfile.zscaler violentutf_api/fastapi_app/Dockerfile ./setup_macos.sh
- Keycloak Authentication Errors: See
/docs/troubleshooting/keycloak-password-fix.md - Docker Build Failures: Ensure Docker Desktop has sufficient resources (8GB RAM minimum)
- Service Connection Issues: Run
./check_services.shto verify all services are running
For more detailed troubleshooting, see the Troubleshooting Guide.
We welcome contributions! Please see our Contributing Guidelines for details.
# Clone repository
git clone https://github.com/cybonto/ViolentUTF.git
# Setup development environment with verbose output
./setup_macos_new.sh --verbose # or setup_linux.sh/setup_windows.bat
# For debugging setup issues
./setup_macos_new.sh --debug
# Activate Python environment
source .vitutf/bin/activate
# Install development dependencies
pip install -r violentutf/requirements.txt- Type Checking: Full mypy compliance
- Code Formatting: Black and isort
- Linting: flake8 and pylint (Note: F541 f-string warnings are intentionally ignored - see code style decisions)
- Testing: pytest with >90% coverage
This project is licensed under the MIT License - see the LICENSE file for details.
ViolentUTF builds upon excellent open-source projects:
- PyRIT - Microsoft's Python Risk Identification Toolkit
- Garak - LLM vulnerability scanner
- APISIX - High-performance API gateway
- Keycloak - Identity and access management
- Streamlit - Python web application framework
- Documentation: docs/ directory
- Issues: GitHub Issues
- Discussions: GitHub Discussions
ViolentUTF - Empowering responsible AI security through comprehensive red-teaming capabilities.