MCP servers are production infrastructure — not developer toys

The Model Context Protocol turned AI assistants from text generators into tool-wielding agents. An MCP server registers tools — functions an agent can invoke — against real systems: source control, databases, cloud consoles, deployment pipelines, analytics platforms, internal APIs. The moment that server accepts a tools/call request, it becomes the boundary between an AI model's intent and an organization's production environment.

Most teams deploy their first MCP server the way they deploy a proof-of-concept: no authentication, no rate limiting, localhost transport, a single run_sql tool that accepts arbitrary queries. The demo works. The security review does not. And the gap between demo and production is not a weekend of polish — it is the same engineering investment required for any service that handles privileged access at machine speed.

MCP server production means treating tool access as an API contract

Every tool registered in an MCP server is an API endpoint with a caller that never gets tired, never reads warnings, and never hesitates before executing. The MCP specification defines a tools/list response that advertises available capabilities and a tools/call request that invokes them. That pair — discovery plus execution — is the same surface area as an API gateway. The operational requirements are identical: authentication, authorization, rate limiting, input validation, audit logging, versioning, and deprecation policy.

The difference is the caller. A human developer hitting an API checks the docs, reads the error message, and adjusts. An agent in a retry loop generates hundreds of requests per minute against the same endpoint. Without server-side rate limiting, a single confused inference cycle can exhaust database connections, burn API quotas, or trigger billing anomalies that take days to trace.

Tool contracts need the same discipline as public API contracts: typed input schemas validated at admission time, bounded output sizes to prevent context-window overflow, idempotency guarantees for mutating operations, and deprecation notices that propagate to the agent host before removal.

Read access and write access are different security products

A tool that searches documentation returns context. A tool that runs a database migration changes state. These two operations share a transport layer but demand entirely different trust models.

The permission tiers that production MCP servers need:

• Observe: read-only access to metrics, logs, and status. No mutation, no sensitive data in responses.
• Suggest: generate a plan, a diff, or a query — but do not execute it. The output requires human review.
• Draft: create a resource in a staging or draft state that has no production side effects until explicitly promoted.
• Execute: apply a change to production. Requires explicit scope grants, caller identity verification, and audit logging.
• Administer: modify the MCP server configuration itself — register tools, change permissions, rotate credentials. Restricted to platform operators, never exposed to agent callers.

Collapsing these tiers into a binary "the agent can call it or it cannot" is the most common design failure. A run_sql tool that accepts arbitrary queries is not a tool — it is an unmonitored shell with database credentials. A run_approved_query tool that accepts a template name and typed parameters is a tool with a bounded blast radius.

Audit trails matter because agents compress time

A human operator deploying a change takes minutes of deliberation, reviews a diff, and types a confirmation. An agent can chain ten tool calls in under a second. When something breaks, the question is not "what happened" — it is "which of the 47 tool calls in the last inference loop caused the side effect, and what arguments did it pass."

Without structured audit logging scoped to the agent session, the incident review is guesswork. Production MCP servers need:

• Per-call logging: agent identity, tool name, input parameters (hashed for sensitive fields), response status, latency, and a correlation ID that stitches the call to the broader agent session.
• Immutable storage: append-only logs that survive the agent session. When SOC 2 or HIPAA reviews arrive, "the agent called the tool but the log rolled over" is not an acceptable answer.
• Alerting on anomalies: repeated calls to the same tool within a single turn (three or more often indicates a retry loop), sudden spikes in error rates per tool, and calls to tools at hours when no human operator is active.

The audit layer is not optional infrastructure added after launch. It is the mechanism that makes autonomous tool access governable. Without it, the first incident costs a weekend of forensics because nobody can answer "who called the tool" with a name.

Narrow tools beat broad shells in every dimension

The temptation when building MCP servers is generality: one execute_command tool that can do anything. This design trades security for convenience and loses both — because a general tool provides no useful context to the agent about what it should do, and no useful constraint about what it must not do.

Narrow tool design follows the principle of least capability:

• One action per tool. create_github_issue is a tool. interact_with_github is a shell.
• Typed, bounded inputs. A deploy_service tool accepts a service name from an enum and an environment from a fixed set — not an arbitrary JSON payload.
• Explicit side effects declared in the tool description. The agent host uses tool descriptions to plan. A description that says "this tool modifies production data" changes the planner's behavior. A vague description like "runs a command" tells the planner nothing.
• Scoped credentials per tool. The tool that reads GitHub issues does not share credentials with the tool that merges pull requests. Credential isolation means a compromised tool path cannot escalate to adjacent capabilities.

This is the same design philosophy behind Unix utilities: small, composable, single-purpose. The blast radius management principles for agentic coding apply with equal force to MCP tool design — every additional capability a tool exposes is attack surface an agent can exercise.

MCP server production requires gateway-level governance at scale

Teams running a single MCP server on localhost can manage auth inline. Teams running five or more servers against production systems need a gateway — the same architectural pattern that API management settled on a decade ago.

An MCP gateway centralizes:

• Authentication: OAuth 2.1 with PKCE for remote servers, verified at the gateway before the request reaches the origin server. Individual servers do not implement their own auth flows.
• Rate limiting on three axes: per user, per tool, and per agent. A server-wide limit of 60 requests per minute fails against agentic loops that legitimately need bursts for multi-step plans.
• Tool registry and discovery: a single tools/list aggregation point that prevents tool-name collisions across servers and enforces RBAC at the tool level.
• Observability: unified metrics, structured logs, and distributed traces across all servers. One Grafana dashboard, not one per server.
• Secret management: credentials stored in a vault, issued as short-lived tokens to the origin server at call time. No long-lived API keys baked into environment variables.

The gateway is not a nice-to-have for mature organizations. Research across production deployments shows that every team running more than three MCP servers in production eventually centralizes through a gateway — or pays the coordination tax of managing per-server auth, per-server monitoring, and per-server secrets indefinitely.

What should an MCP server be allowed to do?

Governance decisions for MCP servers map directly to the risk profile of the tools they expose and the agents they serve.

Should every company maintain an MCP registry?

Any organization with more than three MCP servers needs a registry — a catalog of available servers, their tools, ownership, and permission policies. Without a registry, shadow MCP servers proliferate the same way shadow IT does: an engineer installs a community MCP server with broad filesystem access, connects it to a coding agent, and nobody in security knows it exists until an incident. The internal developer portal approach to agent governance applies here — the portal becomes the registry.

How should credentials flow to an MCP server?

Credentials should never be embedded in the MCP server process. The production pattern is a secrets manager (Vault, AWS Secrets Manager, or the platform's native solution) that issues short-lived, scoped tokens to the server at runtime. Each tool receives only the credentials it needs for its specific operation. When an agent session ends, the tokens expire. This prevents credential accumulation — a single compromised server process cannot access credentials for tools it does not currently serve.

What happens when an MCP tool call fails?

Failed tool calls need three things: a structured error response the agent can interpret (not a raw stack trace), an entry in the audit log with the failure reason, and an alert if the failure rate crosses a threshold. Agents retry by default. If the server does not return a clear, machine-readable error indicating whether the failure is transient or permanent, the agent will retry indefinitely — creating the load spike that the rate limiter should have caught.

The opposing view: strict MCP governance slows adoption

A common argument holds that heavy governance around MCP servers kills the speed advantage of AI agents. If every tool needs a security review, every server needs OAuth 2.1, and every call needs an audit log, teams will avoid building MCP servers entirely — and the agents will remain toy demos that only work against sandboxed environments.

This argument maps to the same debate every security control provokes: the perception that governance and velocity are in tension. In practice, the teams that deploy MCP servers fastest are those with a gateway already in place — because the individual server only needs to implement business logic, not auth, not rate limiting, not observability. The governance infrastructure accelerates server development by removing per-server boilerplate. The slow teams are the ones that skip governance initially, ship quickly, fail a security review, and then spend months retrofitting controls while the servers are already in production carrying real traffic.

Key takeaways

• An MCP server is not a plugin — it is an access layer for autonomous software that requires the same operational rigor as any privileged API.
• Read-only tools and write-capable tools need different trust models; collapsing them into binary access is the most common design failure.
• Audit every tool call with agent identity, parameters, response status, and a session correlation ID — forensics without this data costs weekends.
• Narrow, single-purpose tools with typed inputs and scoped credentials beat general-purpose shells in security, debuggability, and agent performance.
• At three or more MCP servers, a gateway is not optional — centralized auth, rate limiting, and observability pay for themselves immediately.
• Governance infrastructure accelerates MCP adoption; skipping it creates a security debt that compounds at machine speed.

Conclusion

The thing that makes AI agents useful is the same thing that makes them dangerous: tool access. An MCP server that accepts tools/call from an autonomous agent is granting execution capability to software that operates at machine speed, without human hesitation, and without the social context that makes a developer pause before running a destructive command. Treating that server as production infrastructure is not a compliance checkbox — it is the difference between an agent that augments a team and one that becomes an unmonitored attack surface with legitimate credentials. The organizations that gain the most from agentic AI will be those that made the investment in governance before the first incident, not after.