Provider Setup

All providers are configured under providers in config.json. Each provider entry contains a keys array where every key has a name, value, models, and weight, plus optional provider-specific config objects.

In DB-backed deployments, provider entries from config.json are reconciled into the config store at startup. The default source_of_truth: "split" mode preserves UI/API edits while matching file-backed providers are unchanged. With source_of_truth: "config.json", a present providers section is authoritative and prunes DB-only providers or keys. See Source of Truth & Reconciliation.

Supplying credentials: Use the env. prefix to reference environment variables - never put API keys directly in config.json:

{
  "providers": {
    "openai": {
      "keys": [
        {
          "name": "primary",
          "value": "env.OPENAI_API_KEY",
          "models": ["*"],
          "weight": 1.0
        }
      ]
    }
  }
}

Common Provider Fields

Every key object supports these fields:

Field	Type	Description
`name`	string	Unique name for this key (used in logs and virtual key pin)
`value`	string	API key value or `env.VAR_NAME` reference
`models`	array	Models this key serves. `["*"]` = all models
`weight`	float	Load balancing weight. Higher = more traffic
`aliases`	object	Map logical name → actual model name for this key
`use_for_batch_api`	boolean	Mark key as eligible for batch API calls
`oauth_key_config`	object	Mint the bearer from an OAuth token endpoint instead of carrying a static `value`. `openai` and custom providers on the `openai` base only. See below

OAuth-minted keys (`oauth_key_config`)

Leave value empty and set oauth_key_config to have Bifrost request the access token itself, cache it per credential and refresh it before expiry. A non-empty value always wins.

Field	Type	Description
`grant_type`	string	`client_credentials` or `jwt_bearer` (required)
`token_url`	string	Token endpoint, `https` (required; `env.` reference allowed). Plain `http` only for loopback hosts; no userinfo, query or fragment
`client_id`, `client_secret`	string	Client credentials for `client_credentials`
`auth_style`	string	`header` (HTTP Basic, default) or `body`
`private_key`	string	RSA or EC PEM for `jwt_bearer`
`issuer`, `subject`, `audience`, `key_id`	string	Assertion claims and `kid` for `jwt_bearer`; `audience` is an optional form parameter for `client_credentials`
`signing_algorithm`	string	`RS256` (default), `RS384`, `RS512`, `PS256`, `PS384`, `PS512`, `ES256`, `ES384`, `ES512`
`assertion_lifetime_seconds`	integer	Seconds from issue until the assertion expires, default 300, maximum 3600; `iat` is backdated 30 seconds, so `exp - iat` is this value plus 30
`scopes`	array	Scopes requested
`extra_params`	object	Extra form parameters on the token request. Masked on read; encrypted at rest only when `encryption_key` / `BIFROST_ENCRYPTION_KEY` is set. Not for credentials

{
  "providers": {
    "acme-llm": {
      "custom_provider_config": { "base_provider_type": "openai" },
      "network_config": { "base_url": "https://llm.acme.example/v1" },
      "keys": [
        {
          "name": "acme-oauth",
          "models": ["*"],
          "weight": 1.0,
          "oauth_key_config": {
            "grant_type": "client_credentials",
            "token_url": "https://idp.acme.example/oauth2/token",
            "client_id": "env.ACME_CLIENT_ID",
            "client_secret": "env.ACME_CLIENT_SECRET",
            "scopes": ["inference"]
          }
        }
      ]
    }
  }
}

The full field reference with the JWT bearer example is in Custom Providers → Authentication. Per-provider network_config options (applies to all standard providers):

Field	Type	Description
`default_request_timeout_in_seconds`	integer	Per-request timeout. For streaming requests it bounds the connection, request write and wait for response headers; the body is governed by `stream_idle_timeout_in_seconds`
`max_retries`	integer	Retry attempts on transient errors
`retry_backoff_initial`	integer	Initial backoff in milliseconds
`retry_backoff_max`	integer	Maximum backoff in milliseconds
`max_conns_per_host`	integer	Max TCP connections to the provider endpoint (default: 5000)
`extra_headers`	object	Static headers added to every provider request
`stream_idle_timeout_in_seconds`	integer	Idle timeout per stream chunk (default: 120)
`insecure_skip_verify`	boolean	Disable TLS verification (last resort only)
`ca_cert_pem`	string	PEM-encoded CA for self-signed or private CA endpoints

Concurrency and buffering per provider:

Field	Type	Description
`concurrency_and_buffer_size.concurrency`	integer	Max concurrent requests to this provider
`concurrency_and_buffer_size.buffer_size`	integer	Request queue depth

Prompt-cache breakpoint injection per provider (see Auto Prompt Caching):

Field	Type	Description
`prompt_cache.auto_inject`	boolean	Mark the first cacheable block when the caller sent no cache markers. Off by default
`prompt_cache.ttl`	string	Lifetime for injected markers. Only `"1h"` is accepted; omit for the provider default
`prompt_cache.cache_control_injection_points`	array	Target specific messages by `role` and/or `index`. Replaces the first-block strategy when set

OpenAI

Supports multiple keys with weighted load balancing. Mark one key with use_for_batch_api: true to designate it for the Batch API.

{
  "providers": {
    "openai": {
      "keys": [
        {
          "name": "openai-primary",
          "value": "env.OPENAI_KEY_1",
          "models": ["*"],
          "weight": 2.0
        },
        {
          "name": "openai-secondary",
          "value": "env.OPENAI_KEY_2",
          "models": ["gpt-4o-mini"],
          "weight": 1.0
        },
        {
          "name": "openai-batch",
          "value": "env.OPENAI_KEY_BATCH",
          "models": ["*"],
          "weight": 1.0,
          "use_for_batch_api": true
        }
      ],
      "network_config": {
        "default_request_timeout_in_seconds": 120,
        "max_retries": 3,
        "retry_backoff_initial": 500,
        "retry_backoff_max": 5000
      }
    }
  }
}

Anthropic

{
  "providers": {
    "anthropic": {
      "keys": [
        {
          "name": "anthropic-primary",
          "value": "env.ANTHROPIC_KEY_1",
          "models": ["*"],
          "weight": 1.0
        },
        {
          "name": "anthropic-secondary",
          "value": "env.ANTHROPIC_KEY_2",
          "models": ["*"],
          "weight": 1.0
        }
      ],
      "network_config": {
        "default_request_timeout_in_seconds": 180
      }
    }
  }
}

Override Anthropic beta headers (optional):

{
  "providers": {
    "anthropic": {
      "keys": [
        {
          "name": "primary",
          "value": "env.ANTHROPIC_API_KEY",
          "models": ["*"],
          "weight": 1.0
        }
      ],
      "network_config": {
        "beta_header_overrides": {
          "redact-thinking-": true
        }
      }
    }
  }
}

Azure OpenAI

Azure requires azure_key_config on every key with endpoint. Bifrost uses the Azure OpenAI v1 API — no api_version needed. List your Azure deployment names in models - Bifrost routes requests using the model name as the deployment name. If your deployment names differ from the model names you use in requests, add an aliases map on the key.

API Key
Managed Identity / DefaultAzureCredential

{
  "providers": {
    "azure": {
      "keys": [
        {
          "name": "azure-primary",
          "value": "env.AZURE_API_KEY",
          "models": ["gpt-4o", "gpt-4o-mini"],
          "weight": 1.0,
          "azure_key_config": {
            "endpoint": "env.AZURE_ENDPOINT"
          }
        }
      ]
    }
  }
}

Set environment variables:

export AZURE_API_KEY="your-azure-api-key"
export AZURE_ENDPOINT="https://your-resource.openai.azure.com"

When value is empty or omitted, Bifrost uses DefaultAzureCredential - which resolves credentials from Workload Identity, VM managed identity, or az login.

{
  "providers": {
    "azure": {
      "keys": [
        {
          "name": "azure-workload-identity",
          "value": "",
          "models": ["gpt-4o"],
          "weight": 1.0,
          "azure_key_config": {
            "endpoint": "env.AZURE_ENDPOINT"
          }
        }
      ]
    }
  }
}

Deployment name aliases - when your Azure deployment names differ from the model names in requests, use aliases:

{
  "providers": {
    "azure": {
      "keys": [
        {
          "name": "azure-primary",
          "value": "env.AZURE_API_KEY",
          "models": ["gpt-4o"],
          "weight": 1.0,
          "aliases": {
            "gpt-4o": "gpt-4o-prod-deployment"
          },
          "azure_key_config": {
            "endpoint": "env.AZURE_ENDPOINT"
          }
        }
      ]
    }
  }
}

aliases values can also be objects, not just plain strings. The object form lets you tag each alias with a canonical model_name (improves pricing/log attribution when the wire ID is opaque), a model_family for routing, and per-alias provider overrides like api_version or endpoint. See Aliasing Models for the full schema.

Multi-region failover (two keys, different regions):

{
  "providers": {
    "azure": {
      "keys": [
        {
          "name": "eastus",
          "value": "env.AZURE_KEY_EAST",
          "models": ["gpt-4o"],
          "weight": 1.0,
          "azure_key_config": {
            "endpoint": "env.AZURE_ENDPOINT_EAST"
          }
        },
        {
          "name": "westus",
          "value": "env.AZURE_KEY_WEST",
          "models": ["gpt-4o"],
          "weight": 1.0,
          "azure_key_config": {
            "endpoint": "env.AZURE_ENDPOINT_WEST"
          }
        }
      ]
    }
  }
}

AWS Bedrock

Bedrock requires bedrock_key_config with at minimum a region. Three auth modes:

Static Credentials
IAM Role (instance profile / IRSA)
STS AssumeRole

{
  "providers": {
    "bedrock": {
      "keys": [
        {
          "name": "bedrock-static",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "bedrock_key_config": {
            "region": "us-east-1",
            "access_key": "env.AWS_ACCESS_KEY_ID",
            "secret_key": "env.AWS_SECRET_ACCESS_KEY"
          }
        }
      ]
    }
  }
}

When only region is set, Bifrost inherits credentials from the AWS SDK default chain - IRSA (IAM Roles for Service Accounts), EC2 instance profile, or AWS_* env vars.

{
  "providers": {
    "bedrock": {
      "keys": [
        {
          "name": "bedrock-iam",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "bedrock_key_config": {
            "region": "us-east-1"
          }
        }
      ]
    }
  }
}

{
  "providers": {
    "bedrock": {
      "keys": [
        {
          "name": "bedrock-assumerole",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "bedrock_key_config": {
            "region": "us-west-2",
            "role_arn": "env.AWS_ROLE_ARN",
            "external_id": "env.AWS_EXTERNAL_ID",
            "session_name": "bifrost-session"
          }
        }
      ]
    }
  }
}

Model aliases (map logical names to Bedrock inference profile IDs):

{
  "bedrock_key_config": {
    "region": "us-east-1"
  },
  "aliases": {
    "claude-sonnet": "us.anthropic.claude-3-5-sonnet-20241022-v2:0",
    "claude-haiku":  "us.anthropic.claude-3-5-haiku-20241022-v1:0"
  }
}

Batch API - S3 configuration:

{
  "bedrock_key_config": {
    "region": "us-east-1",
    "access_key": "env.AWS_ACCESS_KEY_ID",
    "secret_key": "env.AWS_SECRET_ACCESS_KEY",
    "batch_s3_config": {
      "buckets": [
        {
          "bucket_name": "my-bedrock-batch-bucket",
          "prefix": "batch/",
          "is_default": true
        }
      ]
    }
  }
}

AWS Bedrock Mantle

Bedrock Mantle requires bedrock_mantle_key_config with a required region (no default). It authenticates with AWS SigV4 or an optional Bearer API key. See the Bedrock Mantle provider page for model-ID details.

Static Credentials
IAM Role (instance profile / IRSA)
API Key (Bearer)

{
  "providers": {
    "bedrock_mantle": {
      "keys": [
        {
          "name": "mantle-static",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "bedrock_mantle_key_config": {
            "region": "us-east-1",
            "access_key": "env.AWS_ACCESS_KEY_ID",
            "secret_key": "env.AWS_SECRET_ACCESS_KEY",
            "session_token": "env.AWS_SESSION_TOKEN"
          }
        }
      ]
    }
  }
}

When only region is set, Bifrost inherits credentials from the AWS SDK default chain - IRSA, EC2 instance profile, or AWS_* env vars.

{
  "providers": {
    "bedrock_mantle": {
      "keys": [
        {
          "name": "mantle-iam",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "bedrock_mantle_key_config": {
            "region": "us-east-1"
          }
        }
      ]
    }
  }
}

To assume an IAM role before requests (works with both inherited and explicit credentials), add role_arn (and optionally external_id / session_name):

{
  "bedrock_mantle_key_config": {
    "region": "us-west-2",
    "role_arn": "env.AWS_ROLE_ARN",
    "external_id": "env.AWS_EXTERNAL_ID",
    "session_name": "bifrost-session"
  }
}

Set the top-level value to a Bedrock Mantle API key and leave the SigV4 credentials empty (region is still required).

{
  "providers": {
    "bedrock_mantle": {
      "keys": [
        {
          "name": "mantle-api-key",
          "value": "env.BEDROCK_MANTLE_API_KEY",
          "models": ["*"],
          "weight": 1.0,
          "bedrock_mantle_key_config": {
            "region": "us-east-1"
          }
        }
      ]
    }
  }
}

Google Vertex AI

Vertex requires vertex_key_config with project_id and region. Three auth modes:

Service Account Key
GKE Workload Identity / ADC
AWS Workload Identity (EKS)

{
  "providers": {
    "vertex": {
      "keys": [
        {
          "name": "vertex-sa",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "vertex_key_config": {
            "project_id": "env.VERTEX_PROJECT_ID",
            "region": "us-central1",
            "auth_credentials": "env.VERTEX_AUTH_CREDENTIALS"
          }
        }
      ]
    }
  }
}

VERTEX_AUTH_CREDENTIALS should contain the base64-encoded service account JSON.

When auth_credentials is omitted, Bifrost calls google.FindDefaultCredentials - which resolves to GKE Workload Identity, GCE metadata server, or gcloud auth application-default login.

{
  "providers": {
    "vertex": {
      "keys": [
        {
          "name": "vertex-workload-identity",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "vertex_key_config": {
            "project_id": "my-gcp-project",
            "region": "us-central1"
          }
        }
      ]
    }
  }
}

When Bifrost runs on AWS, aws_workload_identity federates the workload’s IAM role (EKS IRSA / Pod Identity, ECS task role, EC2 instance profile) into a GCP token through a Workload Identity Pool AWS provider. No service account key and no AWS access key are stored. See AWS Workload Identity Federation for the GCP and AWS setup.

{
  "providers": {
    "vertex": {
      "keys": [
        {
          "name": "vertex-eks",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "vertex_key_config": {
            "project_id": "my-gcp-project",
            "region": "us-central1",
            "aws_workload_identity": {
              "audience": "//iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/eks-pool/providers/aws",
              "service_account_email": "bifrost-vertex@my-gcp-project.iam.gserviceaccount.com"
            }
          }
        }
      ]
    }
  }
}

Omit service_account_email to call Vertex AI as the federated principal directly. aws_region and aws_role_arn are optional overrides.

Databricks

Databricks keys carry a workspace URL and, optionally, an OAuth machine-to-machine service principal. Both Databricks inference surfaces are served by this one provider; api_format selects between them, and auto picks by model name.

Personal Access Token
OAuth M2M (Service Principal)

{
  "providers": {
    "databricks": {
      "keys": [
        {
          "name": "databricks-main",
          "value": "env.DATABRICKS_TOKEN",
          "models": ["*"],
          "weight": 1.0,
          "databricks_key_config": {
            "workspace_url": "env.DATABRICKS_WORKSPACE_URL",
            "api_format": "auto"
          }
        }
      ]
    }
  }
}

Leave value unset. Bifrost mints and refreshes tokens from the workspace OIDC endpoint.

{
  "providers": {
    "databricks": {
      "keys": [
        {
          "name": "databricks-service-principal",
          "models": ["*"],
          "weight": 1.0,
          "databricks_key_config": {
            "workspace_url": "env.DATABRICKS_WORKSPACE_URL",
            "api_format": "model_serving",
            "client_id": "env.DATABRICKS_CLIENT_ID",
            "client_secret": "env.DATABRICKS_CLIENT_SECRET",
            "forward_gateway_tags": true
          }
        }
      ]
    }
  }
}

See Databricks for the full field reference.

Standard API-Key Providers

These providers follow the same simple pattern - one or more keys with weights. Replace the provider name and env var name accordingly.

{
  "providers": {
    "groq": {
      "keys": [
        {
          "name": "groq-primary",
          "value": "env.GROQ_API_KEY",
          "models": ["*"],
          "weight": 1.0
        }
      ]
    },
    "gemini": {
      "keys": [
        {
          "name": "gemini-primary",
          "value": "env.GEMINI_API_KEY",
          "models": ["*"],
          "weight": 1.0
        }
      ]
    },
    "mistral": {
      "keys": [
        {
          "name": "mistral-primary",
          "value": "env.MISTRAL_API_KEY",
          "models": ["*"],
          "weight": 1.0
        }
      ]
    },
    "cohere": {
      "keys": [{ "name": "cohere-main", "value": "env.COHERE_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "perplexity": {
      "keys": [{ "name": "perplexity-main", "value": "env.PERPLEXITY_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "xai": {
      "keys": [{ "name": "xai-main", "value": "env.XAI_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "cerebras": {
      "keys": [{ "name": "cerebras-main", "value": "env.CEREBRAS_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "deepseek": {
      "keys": [{ "name": "deepseek-main", "value": "env.DEEPSEEK_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "openrouter": {
      "keys": [{ "name": "openrouter-main", "value": "env.OPENROUTER_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "nebius": {
      "keys": [{ "name": "nebius-main", "value": "env.NEBIUS_API_KEY", "models": ["*"], "weight": 1.0 }]
    }
  }
}

Self-Hosted Providers

Self-hosted providers point to a URL you operate. No API key is typically required ("value": "").

Ollama
vLLM
SGLang
HuggingFace / Replicate

{
  "providers": {
    "ollama": {
      "keys": [
        {
          "name": "ollama-local",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "ollama_key_config": {
            "url": "http://localhost:11434"
          }
        }
      ]
    }
  }
}

Using an env var for the URL (useful across environments):

{
  "ollama_key_config": {
    "url": "env.OLLAMA_URL"
  }
}

vLLM instances are model-specific - one key per served model:

{
  "providers": {
    "vllm": {
      "keys": [
        {
          "name": "vllm-llama3-70b",
          "value": "",
          "models": ["llama-3-70b"],
          "weight": 1.0,
          "vllm_key_config": {
            "url": "http://vllm-server:8000",
            "model_name": "meta-llama/Meta-Llama-3-70B-Instruct"
          }
        },
        {
          "name": "vllm-mistral",
          "value": "",
          "models": ["mistral-7b"],
          "weight": 1.0,
          "vllm_key_config": {
            "url": "http://vllm-mistral:8000",
            "model_name": "mistralai/Mistral-7B-Instruct-v0.3"
          }
        }
      ]
    }
  }
}

{
  "providers": {
    "sgl": {
      "keys": [
        {
          "name": "sgl-main",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "sgl_key_config": {
            "url": "http://sgl-router:30000"
          }
        }
      ]
    }
  }
}

These providers use aliases to map logical model names to provider-specific IDs:

{
  "providers": {
    "huggingface": {
      "keys": [
        {
          "name": "hf-main",
          "value": "env.HF_API_KEY",
          "models": ["llama-3", "mixtral"],
          "weight": 1.0,
          "aliases": {
            "llama-3": "meta-llama/Meta-Llama-3-8B-Instruct",
            "mixtral": "mistralai/Mixtral-8x7B-Instruct-v0.1"
          }
        }
      ]
    },
    "replicate": {
      "keys": [
        {
          "name": "replicate-main",
          "value": "env.REPLICATE_API_KEY",
          "models": ["llama-3"],
          "weight": 1.0,
          "aliases": {
            "llama-3": "meta/meta-llama-3-70b-instruct"
          },
          "replicate_key_config": {
            "use_deployments_endpoint": false
          }
        }
      ]
    }
  }
}

Proxy Configuration

Route provider traffic through an HTTP or SOCKS5 proxy:

{
  "providers": {
    "openai": {
      "keys": [
        { "name": "primary", "value": "env.OPENAI_API_KEY", "models": ["*"], "weight": 1.0 }
      ],
      "proxy_config": {
        "type": "http",
        "url": "http://proxy.corp.example.com:3128",
        "username": "env.PROXY_USER",
        "password": "env.PROXY_PASS"
      }
    }
  }
}

Field	Type	Options
`proxy_config.type`	string	`"none"`, `"http"`, `"socks5"`, `"environment"`
`proxy_config.url`	string	Proxy server URL
`proxy_config.username`	string	Proxy auth username
`proxy_config.password`	string	Proxy auth password (`env.` supported)
`proxy_config.ca_cert_pem`	string	PEM CA for TLS-intercepting proxies

Use "type": "environment" to pick up HTTP_PROXY / HTTPS_PROXY env vars automatically.

Multi-Provider Example

{
  "$schema": "https://www.getbifrost.ai/schema",
  "providers": {
    "openai": {
      "keys": [
        { "name": "openai-primary", "value": "env.OPENAI_API_KEY", "models": ["*"], "weight": 2.0 }
      ]
    },
    "anthropic": {
      "keys": [
        { "name": "anthropic-primary", "value": "env.ANTHROPIC_API_KEY", "models": ["*"], "weight": 1.0 }
      ]
    },
    "groq": {
      "keys": [
        { "name": "groq-primary", "value": "env.GROQ_API_KEY", "models": ["*"], "weight": 1.0 }
      ]
    }
  }
}

With three providers and the weights above, traffic is distributed: 50% OpenAI, 25% Anthropic, 25% Groq. If any provider returns an error, Bifrost automatically retries on the next key or provider.

Managed Kubernetes

Hosted Containers

Config as Code

Enterprise Deployment

Common setup instructions

Common Provider Fields

OAuth-minted keys (`oauth_key_config`)

OpenAI

Anthropic

Azure OpenAI

AWS Bedrock

AWS Bedrock Mantle

Google Vertex AI

Databricks

Standard API-Key Providers

Self-Hosted Providers

Proxy Configuration

Multi-Provider Example

​Common Provider Fields

​OAuth-minted keys (oauth_key_config)

​OpenAI

​Anthropic

​Azure OpenAI

​AWS Bedrock

​AWS Bedrock Mantle

​Google Vertex AI

​Databricks

​Standard API-Key Providers

​Self-Hosted Providers

​Proxy Configuration

​Multi-Provider Example

Common Provider Fields

OAuth-minted keys (`oauth_key_config`)

OpenAI

Anthropic

Azure OpenAI

AWS Bedrock

AWS Bedrock Mantle

Google Vertex AI

Databricks

Standard API-Key Providers

Self-Hosted Providers

Proxy Configuration

Multi-Provider Example