Audit logs for security events vs application logs

Application logs answer why a request failed. Security audit logs answer who changed access, from where, and under which policy. Treating them as one stream produces noisy debugging data and non-defensible audit trails.

Engineering8 min read
Audit logsSecurityObservabilityComplianceLogging
Share

The security review asked for proof that an admin revoked a user's API key on March 12. Engineering exported three terabytes of application logs, grep'd for the user ID, and found four hundred thousand lines of debug output, stack traces, and cache miss messages. The actual revocation event was there — buried in a JSON blob with inconsistent field names, no actor ID, and a timestamp in local server time. The auditor marked the control as failed. Security audit logs and application logs serve different consumers, different retention rules, and different integrity requirements. Merging them does not save money. It guarantees that neither debugging nor investigations work when they matter.

Application logs exist for engineers: reproduce errors, trace latency, inspect state at failure. Security audit logs exist for accountability: demonstrate that authentication, authorization, and sensitive data access followed policy — during an incident, a compliance audit, or a legal hold. The schema, immutability, access controls, and query patterns diverge by design. One stream optimized for both becomes a stream optimized for neither.

Two log types, two contracts

DimensionApplication logsSecurity audit logs
Primary consumerEngineering, on-callSecurity, compliance, legal
Question answeredWhy did this fail?Who did what, when, to what resource?
Typical volumeHigh — debug, info, traceLow — security-relevant events only
RetentionDays to weeks (cost-driven)Months to years (policy-driven)
IntegrityMutable, rotatableAppend-only, tamper-evident
PII handlingOften verbose; may include payloadsMinimal; actor, action, resource, outcome
AccessBroad engineeringRestricted; break-glass audited

Application logs are operational. Audit logs are evidentiary. Evidentiary data requires chain-of-custody thinking from the moment of emission.

Debug logging is not audit logging. A console.log with a user ID is not proof of authorization.

What belongs in a security audit log

Audit events are security-relevant state changes and access decisions — not every HTTP 200.

Authentication events

  • Login success and failure (with reason code, not password)
  • MFA enrollment, challenge, bypass (if policy allows)
  • Session creation, refresh, revocation
  • API key and token issuance, rotation, revocation

Authorization events

  • Access granted or denied to a resource (include policy/scopes evaluated)
  • Role assignment changes
  • Permission elevation (temporary or permanent)
  • OAuth scope grants and denials — which client received which scopes for which resource

Data and configuration events

  • Export or bulk download of sensitive data
  • Admin changes to tenant settings, billing, or security policy
  • Secrets access (not secret values — access to named secret)
  • Infrastructure policy changes affecting trust boundaries

Security control events

  • Firewall rule changes, WAF bypass, break-glass admin activation
  • Failed integrity checks on audit pipeline itself

Each event should carry a stable schema:

audit_event:
  event_id: uuid (idempotent emitter)
  event_type: enum (e.g. auth.login.success)
  timestamp: RFC3339 UTC
  actor:
    type: user | service | api_key
    id: stable identifier
    ip: optional
    user_agent: optional
  resource:
    type: tenant | document | role | ...
    id: stable identifier
  action: create | read | update | delete | grant | deny
  outcome: success | failure
  reason_code: optional (e.g. insufficient_scope)
  request_id: correlation to app logs (not duplication)
  metadata: minimal key-value (no payloads)

request_id links audit to application logs for deep debugging — investigators start in audit, pivot to app logs with correlation, never the reverse as primary.

What should stay out of audit logs

Dumping all application logs into the audit store defeats cost, clarity, and legal scope.

  • Stack traces, query plans, cache keys
  • Full request/response bodies
  • Health check traffic
  • Routine read traffic unless policy mandates data-access logging for regulated data
  • Duplicate events from retries without idempotency keys

High-volume read access logging (every SELECT) belongs in specialized data-access monitoring — not generic audit pipelines — unless regulation explicitly requires it. Volume without investigation workflow is expensive noise.

Application logs remain essential — in their lane

Application logs keep the product running. Structured JSON, correlation IDs, log levels, sampling for high-cardinality debug — standard observability practice. They rotate aggressively. Engineers tail them during incidents. They are not legal records.

Anti-patterns when app logs substitute for audit:

Anti-patternRisk
logger.info("User X deleted Y") onlyNo tamper protection; fields vary by service
Shared index for app + auditEngineers delete indices; retention policies conflict
Logging secrets on errorAudit trail becomes secret leak vector
Client-side-only auditAttacker controls emitted events

Security-sensitive actions must emit audit events server-side at the point of enforcement — after authorization middleware, not before. Client telemetry is supplementary at best.

Integrity, retention, and access control

Audit logs that can be edited by the same admins under investigation fail the basic test. Minimum viable integrity:

Append-only storage — no in-place updates; compensating events for corrections, never silent edits.

Separate write path — application services write to audit sink with distinct credentials; delete permissions absent for application roles.

Immutability window — object locking or WORM storage for compliance retention period.

Clock synchronization — UTC everywhere; NTP monitoring on emitters.

Access logging on audit access — reading audit logs is itself audited for insider-threat and legal-chain requirements.

Retention tiers:

TierDurationStorage classUse case
Hot30–90 daysQuery-optimizedActive investigations
Warm1–3 yearsCheaper searchCompliance audits
Cold7+ years (if required)ArchiveLegal hold

Application log retention stays short — audit retention follows policy documents, not disk price alone.

Authorization design determines audit usefulness

An audit entry access.denied without recording which policy failed forces investigators to guess. OAuth scopes and authorization design should specify not only what clients may request but what the audit layer records when enforcement runs — granted scopes, denied scopes, resource owner, and client ID.

Role-based systems log role changes; attribute-based systems log evaluated attributes. The audit schema must survive permission model evolution — version policy_evaluator_v2 in metadata when rules change, so historical events remain interpretable.

Zero trust for internal APIs assumes every service call is authenticated and authorized. Audit logs make that assumption auditable: service-to-service grants, mTLS identity, and per-route denials become events — not inferred from nginx access logs missing actor context.

Investigation workflow: audit first, app logs second

Typical security incident sequence:

  1. Scope from audit — which actors, resources, time window, success vs deny
  2. Correlate — request_id / trace ID into application logs for technical detail
  3. Contain — revoke sessions, rotate keys (each action emits new audit events)
  4. Report — export audit slice with integrity hashes for legal/compliance

Runbooks should name audit queries by scenario — "suspicious admin activity," "API key abuse," "bulk export" — not require investigators to invent SQL under pressure.

Detection rules on audit streams (impossible travel, privilege escalation velocity, deny-then-success patterns) belong in SIEM or security analytics — separate from application log alerts for error rate.

Testing audit pipelines before auditors do

Audit systems fail quietly — events drop when queues back up, field mappers strip actor IDs during schema changes, and staging environments that disable audit "for speed" drift into production config. Validation belongs in CI and game days, not in compliance questionnaires.

Contract tests per event type — emit synthetic auth.login.failure and assert required fields present in sink within SLA.

Chaos on the audit writer — verify buffer and retry behavior; lost events during deploy are unacceptable.

Replay drills — quarterly exercise: given actor X and date range, produce export in under thirty minutes with integrity attestation.

Schema migration discipline — new fields additive only; renames versioned in event_type or metadata, never silent column swaps that break historical queries.

Application log pipelines tolerate occasional loss; audit pipelines do not. On-call runbooks for the product should include "audit ingestion lagging" as a sev-2 — investigations and legal holds stall when the evidentiary stream gaps.

Compliance mapping without checkbox theater

Frameworks ask for audit trails; they rarely mandate merging them with debug logs. Map controls explicitly:

  • Who → actor block in audit schema
  • What → action + resource
  • When → UTC timestamp with integrity
  • Outcome → success/failure + reason code

Gap analysis is honest: if admin actions are only in application logs, the control is not implemented — relabeling logs does not fix it. Building audit emission alongside new admin features is cheaper than retrofitting after a failed audit.

Privacy regulations affect audit content too: minimize personal data in audit fields, define retention and erasure policy for audit containing user identifiers, document conflicts when erasure requests meet legal hold.

How should teams implement security audit logging?

These decisions prevent audit programs that collapse into expensive log dumps.

Should audit and app logs share the same vendor?

They may share infrastructure — different indices, retention, IAM, and ingestion pipelines. Shared UI is optional; shared write credentials are a liability. Engineers who can delete debug indices must not delete audit archives.

How verbose should denied-access logging be?

Log every authorization denial at audit level for sensitive resources; sample or aggregate high-volume public endpoint denials to prevent cost explosion. Always log denied admin paths — brute force and enumeration show up there first.

Can audit events be async?

Emit after successful commit of the state change — never before. Async delivery is acceptable with durable local buffer; lost audit events are compliance failures. Idempotent event_id handles retries without duplicate rows in investigations.

A common argument runs the other way

The opposing view holds that unified logging reduces cost and complexity — one pipeline, one query language, one bill.

Unified ingestion without separation conflates retention, access, and integrity. Engineers with dashboard access to debug logs gain implicit access to evidentiary data. Short retention on the shared index deletes audit when disk fills. Separation is not duplication — it is contract separation. One vendor can host both streams; one stream cannot host both contracts.

Key takeaways

  • Application logs debug failures; security audit logs prove accountability — different schema, retention, and access.
  • Emit audit events server-side at enforcement with stable actor, resource, action, outcome, and UTC timestamp.
  • Link audit to app logs via request_id — investigate audit first, pivot to app logs for depth.
  • Append-only, restricted-access storage; access to audit logs is itself audited.
  • Authorization design determines audit quality — log policy evaluation, not only allow/deny bit.
  • Test audit pipelines with contract tests, replay drills, and ingestion lag as sev-2 — silent drops fail audits.
  • Compliance maps to audit schema fields — relabeling debug logs is not a control.

Conclusion

Security audit logs are a product of authorization architecture, not a logging afterthought. When revocation, scope grants, and admin changes emit tamper-evident events with stable schema, investigations finish in hours. When those actions scatter across debug output, investigations fail and compliance reviews fail with them.

The exercise: list five admin or security-sensitive actions shipped last quarter. For each, answer: is there a dedicated audit event with actor and outcome? If the only proof is application info logs — close the gap before the next audit or incident, not after.

Related articles

Command Palette

Search for a command to run...