Quick Start: VS Code

Connect the EvalHub MCP server to VS Code with GitHub Copilot in under 5 steps.

Option A: stdio transport (recommended)

1. Install the MCP server

   Download the binary, see Install the MCP server.

2. Create a config file

   mkdir -p ~/.evalhub
   cat > ~/.evalhub/config.yaml <<EOF
   base_url: "https://evalhub.apps.my-cluster.example.com"
   token: "YOUR_TOKEN"
   tenant: "my-team"
   EOF

3. Add the MCP server to VS Code settings

   Open your VS Code settings.json (Ctrl/Cmd+Shift+P > “Preferences: Open User Settings (JSON)”) and add:

   {
     "github.copilot.chat.mcp.servers": {
       "evalhub": {
         "command": "evalhub-mcp",
         "args": ["--config", "~/.evalhub/config.yaml"]
       }
     }
   }

   If using the Python SDK instead of the standalone binary:

   {
     "github.copilot.chat.mcp.servers": {
       "evalhub": {
         "command": "evalhub",
         "args": ["mcp"]
       }
     }
   }

4. Restart VS Code

   Reload the window (Ctrl/Cmd+Shift+P > “Developer: Reload Window”) to pick up the new MCP configuration.

5. Verify the connection

   Open GitHub Copilot Chat and ask:

   @workspace List the available evaluation providers using the evalhub MCP server

Option B: HTTP transport

Use HTTP when the MCP server runs as a shared service.

1. Start the MCP server

   evalhub-mcp --transport http --port 3001 --config ~/.evalhub/config.yaml

2. Add the MCP server to VS Code settings

   {
     "github.copilot.chat.mcp.servers": {
       "evalhub": {
         "type": "http",
         "url": "http://localhost:3001/"
       }
     }
   }

3. Restart VS Code and verify

   Open GitHub Copilot Chat and ask:

   @workspace What benchmarks are available?

Using the MCP Inspector for debugging

The MCP Inspector provides a web UI for testing MCP servers interactively:

npx @modelcontextprotocol/inspector

In the Inspector UI, configure:

• Command: evalhub-mcp
• Arguments: --config ~/.evalhub/config.yaml

Or, with the Python SDK:

• Command: evalhub
• Arguments: --profile agent mcp