Quick Start: VS Code
Connect the EvalHub MCP server to VS Code with GitHub Copilot in under 5 steps.
Option A: stdio transport (recommended)
Section titled “Option A: stdio transport (recommended)”-
Install the MCP server
Follow Install the MCP server to download the standalone binary, or install the Python SDK with MCP support:
Terminal window pip install "eval-hub-sdk[mcp]"The
mcpextra installs theevalhub-mcpbinary used byevalhub mcp run. Ensure bothevalhubandevalhub-mcpare on thePATHavailable to VS Code. If using a virtual environment, activate it before launching VS Code. -
Create a config file
Terminal window mkdir -p ~/.evalhubcat > ~/.evalhub/config.yaml <<EOFbase_url: "https://evalhub.apps.my-cluster.example.com"token: "YOUR_TOKEN"tenant: "my-team"EOF -
Add the MCP server to VS Code settings
Open your VS Code
settings.json(Ctrl/Cmd+Shift+P > “Preferences: Open User Settings (JSON)”) and add:{"github.copilot.chat.mcp.servers": {"evalhub": {"command": "evalhub-mcp","args": ["--config", "~/.evalhub/config.yaml"]}}}If using the Python SDK CLI:
{"github.copilot.chat.mcp.servers": {"evalhub": {"command": "evalhub","args": ["mcp", "run"]}}} -
Restart VS Code
Reload the window (Ctrl/Cmd+Shift+P > “Developer: Reload Window”) to pick up the new MCP configuration.
-
Verify the connection
Open GitHub Copilot Chat and ask:
@workspace List the available evaluation providers using the evalhub MCP server
Option B: HTTP transport
Section titled “Option B: HTTP transport”Use HTTP when the MCP server runs as a shared service.
-
Start the MCP server
Terminal window evalhub-mcp --transport http --port 3001 --config ~/.evalhub/config.yaml -
Add the MCP server to VS Code settings
{"github.copilot.chat.mcp.servers": {"evalhub": {"type": "http","url": "http://localhost:3001/"}}} -
Restart VS Code and verify
Open GitHub Copilot Chat and ask:
@workspace What benchmarks are available?
Using the MCP Inspector for debugging
Section titled “Using the MCP Inspector for debugging”The MCP Inspector provides a web UI for testing MCP servers interactively:
npx @modelcontextprotocol/inspectorIn the Inspector UI, configure:
- Command:
evalhub-mcp - Arguments:
--config ~/.evalhub/config.yaml
Or, with the Python SDK:
- Command:
evalhub - Arguments:
--profile agent mcp