MultiLLM

A powerful application that provides a unified frontend for multiple Language Learning Models (LLMs), including OpenAI's Realtime API, Claude, and local models through Ollama.

Features

• Multiple LLM Support:

  • OpenAI GPT-4 (including Realtime API with streaming)
  • Anthropic Claude
  • Local models via Ollama
  • Easy to extend for other LLMs

• Advanced Capabilities:

  • Image Generation
  • Web Search Integration
  • Weather Information
  • Calendar Integration
  • Email Search
  • Math Problem Solving
  • Research Assistant

• Robust Architecture:

  • Persistent WebSocket connections with automatic reconnection
  • Session management and conversation history
  • Tool orchestration and management
  • Streaming responses for real-time interaction

Setup Environment

# Create and activate virtual environment
python -m venv venv
.\venv\Scripts\activate  # Windows
source venv/bin/activate  # Linux/Mac

# Install dependencies
pip install -r requirements.txt

Configuration

1. Copy and rename configuration files:

cp users_example.json users.json
cp llm_config_example.json llm_config.json
cp environment_example.env environment.env

2. Configure the application:

• users.json: Define local users for the application

• llm_config.json: Configure LLM settings:

  {
    "default": "openai_realtime",  // Default LLM to use
    "llms": {
      "openai_realtime": {
        "name": "OpenAI Realtime",
        "model": "gpt-4o-realtime-preview-2024-12-17",
        "type": "chat",
        "info_link": "https://platform.openai.com/docs",
        "wait_limit": 300,
        "enabled": true
      },
      "openai": {
        "name": "OpenAI GPT-4",
        "model": "gpt-4o",
        "type": "chat",
        "info_link": "https://platform.openai.com/docs",
        "wait_limit": 300,
        "enabled": true
      },
      "claude": {
        "name": "Claude",
        "model": "claude-3-opus-20240229",
        "type": "chat",
        "info_link": "https://docs.anthropic.com/claude/docs",
        "wait_limit": 300,
        "enabled": true
      },
      "ollama": {
        "name": "Ollama",
        "model": "llama2",
        "type": "chat",
        "info_link": "https://ollama.ai/library",
        "wait_limit": 300,
        "enabled": true
      }
    }
  }

  Configuration options:

  • default: The default LLM to use when starting a conversation
  • llms: Object containing configurations for each LLM:
    • name: Display name for the LLM
    • model: Model identifier to use
    • type: Type of interaction ('chat', 'assistant', or 'image')
    • info_link: Documentation link for the LLM
    • wait_limit: Maximum time to wait for responses (in seconds)
    • enabled: Whether this LLM is available for use

• environment.env: Set up environment variables:

  # Server Configuration
  APP_IP=127.0.0.1  # Change for external access
  APP_PORT=5000
  SECRET_KEY=your_secret_key
  
  # OpenAI Configuration
  OPENAI_API_KEY=your_openai_key
  OPENAI_ORG_ID=your_org_id  # Optional
  
  # Claude Configuration
  CLAUDE_API_KEY=your_claude_key
  
  # Ollama Configuration
  OLLAMA_API_BASE_URL=http://localhost:11434  # Modify if not using local install
  
  # Additional Services
  GOOGLE_API_KEY=your_google_key  # For web search
  GOOGLE_CX=your_google_cx
  OPENWEATHERMAP_API_KEY=your_weather_key
  

Running the Application

python main.py

The application will be available at http://localhost:5000 (or your configured IP/port).

Architecture

The application uses a modular architecture with several key components:

• OpenaiRealtime: Handles real-time streaming connections with OpenAI's API
• OpenaiRealOrchestrator: Manages tool integration and orchestration
• Multiple Agents: Specialized agents for different tasks (web search, email, calendar, etc.)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.