---

Quick Definition (30–60 words)

Pre-apply checks are automated validations run before configuration or infrastructure changes are applied to live environments. Analogy: like pre-flight checks a pilot does before takeoff. Formal: a gating stage in CI/CD that verifies policy, drift, compatibility, security, and observability prerequisites to reduce risk.

---

What is pre-apply checks?

Pre-apply checks are automated gates that run immediately before a change is applied to infrastructure, configuration, or deployments. They are not general CI tests, nor are they post-deploy monitors. They execute in the narrow window between “ready-to-deploy” and “apply/deploy”, preventing dangerous changes from reaching production.

Key properties and constraints:

• Time-bounded: must complete quickly to avoid blocking pipelines.
• Deterministic where possible: flaky checks cause friction.
• Observable: outputs must feed dashboards and audit logs.
• Remediable: provide clear remediation steps or automated rollback hooks.
• Policy-aware: enforce security and compliance as code.

Where it fits in modern cloud/SRE workflows:

• After static analysis, unit/integration tests, and peer review.
• As a final gate in CI/CD pipelines, pre-merge for infra-as-code, or pre-apply for mutable systems.
• Integrated with policy engines, drift detectors, canary controllers, and service meshes.

Text-only diagram description readers can visualize:

• Developer commits code -> CI runs tests -> Merge to main -> Pre-apply checks execute -> If pass then Apply/Deploy to staging or production -> Observability and canary monitor -> Roll forward or rollback.

pre-apply checks in one sentence

A fast, automated gate that validates configuration, policy, compatibility, and runtime expectations immediately before applying changes to infrastructure or services.

pre-apply checks vs related terms (TABLE REQUIRED)

ID	Term	How it differs from pre-apply checks	Common confusion
T1	CI tests	Runs earlier and focuses on code correctness	People assume CI covers infra policies
T2	Post-deploy monitoring	Runs after change; detects runtime issues	Confused as a replacement for pre-checks
T3	Policy-as-code	One input to pre-apply checks	Assumed to be the entire pre-apply system
T4	Drift detection	Detects differences post-fact	Thought to prevent bad apply proactively
T5	Admission controller	In-cluster blocker at runtime	Mistaken for pipeline pre-apply
T6	Canary analysis	Observes behavior after partial rollout	Believed to be pre-apply safety net
T7	Static analysis	Code/config linting before apply	People expect it to catch runtime issues
T8	Feature flags	Control runtime behavior post-deploy	Misused as a substitute for pre-apply validation

Row Details (only if any cell says “See details below”)

• None

---

Why does pre-apply checks matter?

Business impact:

• Reduces revenue risk by preventing regressions and outages that cause downtime or incorrect behavior.
• Preserves customer trust by reducing visible incidents and rollbacks.
• Reduces compliance fines and audit findings by enforcing policy before change.

Engineering impact:

• Lowers incident frequency by catching risky changes earlier in the pipeline.
• Increases deployment velocity by automating gate decisions and reducing manual review toil.
• Improves developer confidence to ship frequently with smaller blast radius.

SRE framing:

• SLIs/SLOs: pre-apply checks indirectly affect service availability and correctness by preventing bad changes.
• Error budget: effective pre-apply checks reduce burned error budget and make safe releases more predictable.
• Toil: automation via pre-apply checks reduces manual verification and repetitive checks on-call staff perform.
• On-call: fewer emergency rollbacks and less noisy alert churn, but on-call must own remediation actions for failed checks that block releases.

3–5 realistic “what breaks in production” examples:

• Misconfigured network security group opens unintended ports, allowing public access to internal services.
• Database schema migration that performs a full table rewrite causing long locks and high CPU, stalling queries.
• IAM policy change that revokes service account permissions and causes cascading service failures.
• Autoscaler misconfiguration that reduces replica counts below safe thresholds.
• A new feature toggled on by default that sends increased event volume to a third-party API causing rate-limit failures.

---

Where is pre-apply checks used? (TABLE REQUIRED)

ID	Layer/Area	How pre-apply checks appears	Typical telemetry	Common tools
L1	Edge and network	Validate firewall and CDN config before apply	config diff, deploy time	policy engines
L2	Cluster orchestration	Verify kube manifests, admission policies	dry-run results, pod spec checks	kubectl, admission
L3	Service and app	Lint manifests and compatibility tests	lint output, test pass rate	linters, unit tests
L4	Data and database	Migration dry-run and cost estimate checks	migration time estimate	migration tools
L5	Cloud infra (IaaS)	Plan validation and cost check	infra plan delta, cost delta	infra planers
L6	Serverless and PaaS	Cold-start and config validation	invocation simulation	serverless test tools
L7	CI/CD pipelines	Final gating step before apply	gate pass/fail metrics	pipeline systems
L8	Security & compliance	Policy enforcement and scanner results	compliance pass rate	scanning tools
L9	Observability	Validate telemetry instrumented before deploy	metric presence check	observability checks
L10	Incident response	Verify runbook hooks and rollback paths	runbook completeness	runbook tests

Row Details (only if needed)

• None

---

When should you use pre-apply checks?

When it’s necessary:

• High-impact systems where failures cause revenue or safety loss.
• Infrastructure-as-code for production environments.
• Changes touching security, network, IAM, or critical stateful services.
• Migrations altering schemas or data stores.

When it’s optional:

• Small cosmetic changes with zero runtime impact.
• Internal development sandboxes where speed matters more than safety.
• Rapid prototyping where reverts are acceptable.

When NOT to use / overuse it:

• Do not block developer flow for trivial cosmetic changes.
• Avoid adding slow or flaky checks that delay delivery and encourage bypass.
• Don’t replicate every test; keep checks focused and fast.

Decision checklist:

• If change affects security or availability AND affects production -> enforce pre-apply checks.
• If change is low-impact AND isolated to a dev sandbox -> optional fast checks.
• If latency of the check > acceptable pipeline delay -> move to early CI or shift to post-deploy monitoring.

Maturity ladder:

• Beginner: Basic linting and terraform plan validation; fast and manual overrides.
• Intermediate: Policy-as-code, dry-run execution, basic automated remediations.
• Advanced: Full environment simulation, cost estimation, canary orchestration hookup, ML-assisted anomaly prediction, automated rollback.

---

How does pre-apply checks work?

Components and workflow:

1. Trigger: pipeline stage or manual action triggers pre-apply checks.
2. Context collection: gather target environment state, current manifests, version metadata.
3. Static validation: linting, schema and types checks, policy-as-code evaluation.
4. Dynamic dry-run: plan/apply dry-run, simulated deployment, dependency checks.
5. Safety checks: resource quotas, cost delta, permission changes, migration safety.
6. Observability validation: ensure new metrics/logs/traces are instrumented and shipping.
7. Decision engine: combine checks into pass/fail verdict with risk score.
8. Action: approve auto-apply, block and require manual remediation, or auto-fix and re-run.

Data flow and lifecycle:

• Source code/infra repo -> CI pipeline -> pre-apply checks read repo + environment state -> compute results -> persist results to audit log + signal pipeline -> apply or block.

Edge cases and failure modes:

• Intermittent upstream APIs cause dry-run failures.
• Configuration drift between environment snapshot and actual runtime leads to false positives.
• Long-running checks block release windows; need timeouts and fallbacks.
• Overly permissive autofix changes create unreviewed behavior drifts.

Typical architecture patterns for pre-apply checks

1. Pipeline-hooked pre-apply: checks run as a CI job immediately before apply; use when you control pipeline end-to-end.
2. Agent-based environment validator: a small agent queries runtime state and returns validation; use when runtime context is necessary.
3. Simulation sandbox: create ephemeral environment to run a full apply simulation; use for high-risk migrations and complex infrastructure.
4. Policy engine gate: external policy-as-code service evaluates change diffs via webhooks; use for compliance-centralized organizations.
5. Observability-instrumentation check: tests that necessary telemetry exists and will ship; use for teams with strict SLOs.
6. Hybrid: combine dry-run plus canary orchestration to allow safe auto-apply flows.

Failure modes & mitigation (TABLE REQUIRED)

ID	Failure mode	Symptom	Likely cause	Mitigation	Observability signal
F1	Flaky external call	Intermittent gate failures	Upstream API instability	Retry with backoff and cache	spike in gate failures
F2	Stale environment snapshot	False positives on diff	Out-of-date state pulled	Use live queries or refresh	mismatch rate metric
F3	Slow checks	Pipeline timeouts	Heavy simulation or tests	Timeout and degrade to warning	increased pipeline duration
F4	Overly strict policy	Frequent blocks and bypass	Rules too rigid	Review and relax rules	high override count
F5	Autofix regression	Unexpected behavior post-fix	Unreviewed auto changes	Require review for autofix	increase in reverts
F6	Missing telemetry validation	Metrics absent after deploy	Instrumentation not added	Fail deploy or auto-revert	missing metric alerts

Row Details (only if needed)

• None

---

Key Concepts, Keywords & Terminology for pre-apply checks

Note: brief definitions; each line: Term — definition — why it matters — common pitfall

Acceptance testing — Automated final-stage tests validating change behavior — Ensures functional correctness before apply — Too slow to run as a pre-apply gate Admission controller — In-cluster runtime gate for Kubernetes changes — Prevents harmful resources from being created — Confused with pipeline pre-apply checks Air-gapped validation — Checks run without external network access — Required for high-security environments — Hard to simulate real runtime Audit log — Immutable record of check results and decisions — Required for compliance and forensics — Often not centralized Autofix — Automatic remediation applied when a check fails — Reduces manual toil — Can introduce unexpected changes Authority model — Who can bypass or approve gates — Controls risk and accountability — Weak models lead to risky overrides Baseline metrics — Expected metric ranges used for validation — Detects abnormal behavior early — Poor baselines cause false positives Canary analysis — Gradual rollout with automated validation — Limits blast radius after apply — Not a replacement for pre-apply checks Chaos testing — Intentional fault injection to test resilience — Confirms pre-apply assumptions under failure — Not suitable as a primary gate Change window — Allowed time to change production — Limits when heavy checks run — Missing windows cause delays CI pipeline — System orchestrating automated checks and deploys — Hosts the pre-apply stage — Overloaded pipelines slow teams Compatibility matrix — Supported versions and dependencies list — Prevents incompatibility at deploy time — Often out of date Cost estimation — Predicts cost delta for infra changes — Prevents surprise bills — Hard to be precise for dynamic workloads Credential validation — Ensures secrets and permissions are correct — Avoids permission-related failures — Leaking creds is a security risk Data migration dry-run — Simulate migration without applying to production — Finds locking and duration issues — Difficult at large scale Decision engine — Aggregates check outputs into actions — Standardizes gating logic — Complex rules become opaque Declarative infra — Describing desired state rather than imperative steps — Enables dry-run and plan comparisons — Divergence can be confusing Deployment plan — Detailed steps to apply changes — Used to validate and preview changes — Often missing in quick deploys Diff analysis — Comparing desired and current state — Surface risky operations before change — Large diffs need special handling Drift detection — Identify divergence between declared and actual state — Important for long-lived infra — Noisy without thresholds Dry-run/apply plan — Simulation of the apply operation — Reveals destructive ops before execution — Some providers have limited dry-run semantics Feature flagging — Toggle features without deploys — Reduces risk of new code — Misuse hides necessary pre-apply checks GitOps — Declarative, repo-driven operations model — Integrates pre-apply with pull requests — Delays if sync loops are slow Immutable infrastructure — Replace instead of modify pattern — Simplifies reasoning for pre-apply checks — Higher cost for small changes Instrumentation check — Ensures the code emits required telemetry — Critical for observability and SLOs — Too strict checks break agility Integration test — Tests cross-service interactions — Detects systemic regressions — Typically too slow for pre-apply unless scoped Issue tracking link — Associate checks to tickets and runbooks — Improves traceability — Missing links reduce follow-through Kubernetes dry-run — Kube API dry-run simulation for manifests — Useful quick validation — Not comprehensive for runtime failures Latency budget — Allowable latency in checks to avoid blocking — Balances safety and velocity — Often underestimated Manifest linting — Syntax and best-practice validation for manifests — Catch common mistakes early — Lint rules too strict block dev flow Migration safety checks — Verify that migrations won’t harm availability — Protects data integrity — Hard to model for complex schemas Observability completeness — Metric/log/trace presence and labels — Enables post-deploy debugging — Overlooked in many releases On-call playbook — Operational steps for failed checks or blocked deploys — Reduces response time — Outdated playbooks cause delays Policy-as-code — Policy expressed in executable rules — Automates compliance gating — Rule proliferation is a management issue Prereq verification — Check for external dependencies and quotas — Avoids runtime surprises — Often skipped for speed Rollback plan — Predefined steps to revert a change — Essential safety net — Unclear rollback causes confusion during incidents Runbook automated tests — Regular validation of runbook steps against live systems — Ensures runbooks are actionable — Time-consuming to maintain Sanity checks — Lightweight checks to detect obviously bad changes — Fast and effective early blocker — Over-reliance prevents deeper testing Security scanner — Static or dynamic check for vulnerabilities — Prevents known issue deploys — False positives need triage Service-level indicator — Measurable signal of service health — Ties pre-apply to SLOs — Choosing the wrong SLI misleads teams Slack/notification gating — Inform or require approval via chatops — Improves human oversight — Chat noise leads to missed approvals Synthetic test — Programmed external tests that mimic user traffic — Validates real-world behavior — Flaky networks cause false alarms Validation harness — Framework to run pre-apply checks consistently — Standardizes checks across teams — Can become a bottleneck Version matrix — Supported software and infra versions — Prevents unsupported combos — Poor maintenance reduces value Whitelist/blacklist rules — Quick allow or deny patterns for changes — Fast decisions for known-safe items — Overly broad lists create risk YAML schema validation — Ensures manifest structure correctness — Catches structural errors early — Schema drift reduces usefulness Zero-downtime check — Validate that change will not disrupt traffic — Protects availability — Hard for stateful systems

---

How to Measure pre-apply checks (Metrics, SLIs, SLOs) (TABLE REQUIRED)

ID	Metric/SLI	What it tells you	How to measure	Starting target	Gotchas
M1	Gate pass rate	Percentage of changes passing pre-apply	pass count divided by total attempts	95%	High pass hides unvalidated risky changes
M2	Mean gate duration	Time to complete checks	avg duration of gate runs	< 2 min	Long checks slow delivery
M3	Override rate	Rate of manual bypass events	overrides divided by total gates	< 1%	High override means rules unusable
M4	False positive rate	Valid change blocked incorrectly	blocked then later allowed	< 2%	Hard to label without human review
M5	Deployment failure after pass	Failures post-apply despite gate pass	failed deploys divided by passes	< 0.5%	Indicates gap in checks
M6	Time to remediate failed gate	Time from fail to fix	avg time to resolution	< 1 hour	Long times block launches
M7	Cost delta accuracy	Accuracy of predicted vs actual cost	predicted vs actual % diff	within 15%	Cloud cost variability
M8	Telemetry coverage	Percentage of changes with required metrics	count with metrics / total	100% for critical services	Hard to auto-verify certain metrics
M9	Policy violation rate	Frequency of policy infractions	violations per change	0 for critical policies	Noise from low-severity rules
M10	Audit trace completeness	Audit entries per gate run	entries logged per attempt	100%	Missing logs weaken compliance

Row Details (only if needed)

• None

Best tools to measure pre-apply checks

Tool — Prometheus

• What it measures for pre-apply checks: Gate durations, pass/fail counters, override rates
• Best-fit environment: Kubernetes and cloud-native stacks
• Setup outline:
• Export metrics from pre-apply service
• Configure scrape targets and relabel rules
• Create recording rules for SLIs
• Alert on SLO burn rates
• Strengths:
• Powerful query language and ecosystem
• Works well with Kubernetes
• Limitations:
• Not ideal for long-term high-cardinality metrics
• Requires operational overhead

Tool — Grafana

• What it measures for pre-apply checks: Dashboards for SLIs and drilldowns
• Best-fit environment: Any environment storing metrics/logs
• Setup outline:
• Connect Prometheus and logging backends
• Build executive and on-call dashboards
• Configure alerting via Grafana Alerting
• Strengths:
• Flexible visualization and alerting
• Good templating for teams
• Limitations:
• Dashboards require maintenance
• Users may create fragmented views

Tool — Open Policy Agent (OPA)

• What it measures for pre-apply checks: Policy decision logs and violation counts
• Best-fit environment: Cloud-native, Kubernetes, CI
• Setup outline:
• Author policies as Rego
• Integrate OPA with CI and admission flows
• Log decisions to observability backend
• Strengths:
• Flexible and expressive policy language
• Widely adopted
• Limitations:
• Rego learning curve
• Performance tuning needed at scale

Tool — Terraform Cloud / Enterprise

• What it measures for pre-apply checks: Plan diffs, cost estimates, policy checks
• Best-fit environment: Teams using Terraform for infra
• Setup outline:
• Use plan and policy checks as pre-apply gates
• Collect run metrics and decision logs
• Integrate with VCS and CI
• Strengths:
• Built-in plan review workflow
• Policy enforcement and governance
• Limitations:
• Requires Terraform usage
• Enterprise features may be needed for org-wide policies

Tool — Policy-as-code scanner (generic)

• What it measures for pre-apply checks: Rule violations on manifests and configs
• Best-fit environment: Multi-cloud, hybrid infra
• Setup outline:
• Plug into pipeline as a job
• Configure rules and severity levels
• Emit structured results to logs and metrics
• Strengths:
• Fast checks that integrate easily
• Limitations:
• Rule maintenance burden
• Potential false positives

Tool — Synthetic test runner

• What it measures for pre-apply checks: End-to-end behavior of critical flows
• Best-fit environment: Services with stable APIs
• Setup outline:
• Record critical flows
• Run lightweight simulations against staging
• Fail gate if regressions observed
• Strengths:
• Realistic validation
• Limitations:
• Can be flaky on environment variability
• Not suitable for heavy loads

Recommended dashboards & alerts for pre-apply checks

Executive dashboard:

• Panels:
• Gate pass rate trend: shows health of release gating
• Policy violation heatmap: high-level risk areas
• Average time to remediate broken gates: operational efficiency
• Number of overrides and by approver: governance signal
• Why: Stakeholders need high-level safety and throughput metrics

On-call dashboard:

• Panels:
• Active blocked changes: queue of blocked deploys with owners
• Failed gates by type: fast triage of blocking reasons
• Recent failures with logs and links: reduce time to remediate
• Gate duration and pipeline backlog: detect systemic slowdowns
• Why: Enable rapid action to unblock critical deploys

Debug dashboard:

• Panels:
• Per-check granular logs and timing breakdown
• Last N diffs and dry-run outputs
• Metric coverage per service and missing metrics list
• Decision engine score components for a change
• Why: Deep debugging for engineers fixing failing checks

Alerting guidance:

• What should page vs ticket:
• Page: gate failures affecting production releases or multiple services, or systemic gate outages.
• Ticket: single low-impact lint failures, cost-estimate warnings, or advisory policy warnings.
• Burn-rate guidance:
• Alert on error budget burn for deployment failures when >50% of allowed budget consumed in 24 hours.
• Noise reduction tactics:
• Dedupe alerts by change ID and service
• Group low-severity policy violations into daily digest
• Suppress repeated identical failures until acknowledged

---

Implementation Guide (Step-by-step)

1) Prerequisites – Source control and CI pipeline in place. – Inventory of critical services and their SLIs. – Policy definitions and ownership for infra areas. – Logging, metrics, and trace backends available.

2) Instrumentation plan – Define required telemetry per service. – Add lightweight metric emission for gate results. – Ensure dry-run and decision logs are structured and sent to central store.

3) Data collection – Collect plan diffs, dry-run outputs, policy decisions, and telemetry validation results. – Store artifacts with change IDs for audit and debugging.

4) SLO design – Choose SLIs from the measurement table. – Set realistic SLOs and error budgets for gating reliability and remediation times.

5) Dashboards – Create executive, on-call, and debug dashboards described earlier. – Add drilldowns from high-level metrics to individual check artifacts.

6) Alerts & routing – Configure alerts for paging and ticketing rules. – Route paging to the owner of the gate service and a secondary platform on-call.

7) Runbooks & automation – Create runbooks for common failures with exact commands and escalation paths. – Automate common remediations if safe and reversible.

8) Validation (load/chaos/game days) – Run game days that simulate failing pre-apply checks and blocked deploys. – Validate that remediation paths and runbooks work.

9) Continuous improvement – Track override and false-positive rates. – Iterate on rules to reduce noise and speed up checks.

Pre-production checklist:

• Linting and schema validation passes locally.
• Dry-run matches expected plan with no destructive operations.
• Telemetry checks confirm required metrics exist.
• Cost impact estimated and within acceptable bounds.
• Backup and rollback plan documented.

Production readiness checklist:

• Decision engine integrated with CI and audit logs enabled.
• Alerts and runbooks validated on-call.
• Thresholds and SLOs configured and monitored.
• Cross-team signoff for high-impact changes.

Incident checklist specific to pre-apply checks:

• Capture gate failure artifacts and change ID.
• Notify owner and on-call with links to logs and diffs.
• Execute runbook steps to remediate or rollback.
• Record time to resolution and update ticket.
• Postmortem if error budget burned or production impacted.

---

Use Cases of pre-apply checks

1) Network ACL changes – Context: Changing firewall rules in prod. – Problem: Misopen ports expose services. – Why pre-apply helps: Validates diff and runs simulation against rules. – What to measure: Gate pass rate, override count. – Typical tools: Policy engine, dry-run firewall simulator.

2) Database schema migration – Context: Rolling out schema changes. – Problem: Locking and long migration times. – Why pre-apply helps: Dry-run migration and estimate time. – What to measure: Migration time estimate accuracy. – Typical tools: Migration tools with dry-run, backups.

3) IAM policy changes – Context: Modifying service roles. – Problem: Service breakage due to revoked permissions. – Why pre-apply helps: Detects permission removals and dependency checks. – What to measure: Post-deploy failure rate. – Typical tools: IAM differs, static analyzers.

4) Autoscaler configuration update – Context: Tweaking HPA or autoscaling rules. – Problem: Underprovisioning or runaway autoscaling costs. – Why pre-apply helps: Validate min/max and test scaling logic. – What to measure: Post-deploy latency and replica counts. – Typical tools: Kubectl dry-run, canary controllers.

5) Third-party API integration – Context: Changing rate or endpoints to external APIs. – Problem: Rate limiting and unexpected cost. – Why pre-apply helps: Validate expected request patterns and quotas. – What to measure: Synthetic test success rate. – Typical tools: Synthetic runners and API contract tests.

6) Feature flag defaults – Context: New flags defaulting to on. – Problem: Unexpected traffic patterns. – Why pre-apply helps: Validate configuration and default state across environments. – What to measure: Override rates and user impact metrics. – Typical tools: Feature-flag platforms, config lint.

7) Cost controls on infra – Context: Big instance type changes. – Problem: Sudden cost increase. – Why pre-apply helps: Cost delta estimation and alerts. – What to measure: Predicted vs actual cost delta. – Typical tools: Cost estimation tools.

8) Observability changes – Context: Adding new services requiring telemetry. – Problem: Poor diagnosing ability after deploy. – Why pre-apply helps: Verify instrumentation and label adherence. – What to measure: Telemetry coverage percentage. – Typical tools: Telemetry linter and synthetic tests.

9) Canary rollouts – Context: Progressive deploy of new version. – Problem: Rapid reversal hard without prior validation. – Why pre-apply helps: Validates canary configuration and traffic routing rules. – What to measure: Canary success rate and rollback frequency. – Typical tools: Canary analysis platforms.

10) Regulatory compliance change – Context: Data residency or encryption updates. – Problem: Non-compliant configs in production. – Why pre-apply helps: Policy enforcement pre-apply. – What to measure: Policy violation rate. – Typical tools: Policy-as-code engines.

---

Scenario Examples (Realistic, End-to-End)

Scenario #1 — Kubernetes deployment with config drift prevention

Context: A microservice team prepares a manifest update that modifies resource limits and adds an init container.
Goal: Prevent resource regressions and ensure the new init container runs correctly.
Why pre-apply checks matters here: Kubernetes manifests are easy to misconfigure; a bad limits config or failing init container can cause outages.
Architecture / workflow: GitOps repo -> CI -> pre-apply checks job -> kube dry-run + admission policy check + telemetry validation -> apply via GitOps operator.
Step-by-step implementation:

• Add lint and schema validation for manifest.
• Run kubectl apply –dry-run=server against a live API server.
• Run a container image scan and init-container startup simulation in a sandbox.
• Verify expected metrics exist post-deploy (synthetic).
• Gate decision combines checks; only successful changes merge to main.
  What to measure: Gate pass rate, mean gate duration, telemetry coverage.
  Tools to use and why: kubectl dry-run for quick validation, OPA for policy checks, Prometheus for metrics.
  Common pitfalls: Dry-run differences across K8s versions; stale CRD schema leads to false failures.
  Validation: Run a game day that introduces a misconfigured limit and observe blocked deploy.
  Outcome: Reduced incidents related to misconfigured pod specs.

Scenario #2 — Serverless function permission change

Context: A function’s IAM role is tightened to remove S3 write permission.
Goal: Ensure no dependent services fail after permission tightening.
Why pre-apply checks matters here: IAM mistakes are common and cause silent failures.
Architecture / workflow: VCS PR -> CI -> IAM diff checker -> permission dependency analysis -> simulated invocation -> gate decision.
Step-by-step implementation:

• Compute IAM policy diff and list services/accounts referencing role.
• Run a simulated function invocation with mocked downstream services.
• Fail gate if dependent call patterns include S3 writes.
• Require manual approval if impact is non-local.
  What to measure: Override rate, post-deploy error incidents.
  Tools to use and why: IAM diff tooling, local invocation harness, policy engine.
  Common pitfalls: Complex cross-account references are hard to detect.
  Validation: Run a staged deploy to a canary function and observe blocked attempt.
  Outcome: Zero production permission regressions for this change class.

Scenario #3 — Incident-response: blocked deploy during outage

Context: A deployment is blocked by a pre-apply check during an active incident.
Goal: Rapidly determine whether to unblock for rollback or keep blocked to preserve safety.
Why pre-apply checks matters here: During incidents, blocked deploys may be necessary but also can delay rollback fixes.
Architecture / workflow: CI blocked -> incident channel notifies on-call -> decision via runbook.
Step-by-step implementation:

• On-call consults runbook that lists criteria for safe override.
• If rollback required, run a validated rollback that has been pre-approved by checks.
• Log override and create postmortem ticket.
  What to measure: Time to remediate, override audit trail.
  Tools to use and why: Chatops approval, audit logs, runbook tests.
  Common pitfalls: Overrides without follow-up postmortem.
  Validation: Simulate an incident requiring override and ensure runbook remains effective.
  Outcome: Faster incident resolution with auditability.

Scenario #4 — Cost/performance trade-off for instance type change

Context: Team considers switching instance family to reduce cost but wants to avoid performance regressions.
Goal: Validate cost estimate and ensure latency SLOs remain met.
Why pre-apply checks matters here: Cost decisions can degrade performance if underprovisioned.
Architecture / workflow: Infra change PR -> cost estimator + performance simulation -> load synthetic tests in staging -> pre-apply gate.
Step-by-step implementation:

• Run cost estimation for proposed instance type.
• Run performance-sensitive synthetic tests simulating peak traffic.
• Validate SLO adherence and estimated cost savings.
• Gate fails if latency SLO would be violated.
  What to measure: Predicted vs actual cost, SLI latency under load.
  Tools to use and why: Cost tools, synthetic load test runner, observability stack.
  Common pitfalls: Synthetic tests not matching production traffic patterns.
  Validation: Blue-green rollout with small percentage canary.
  Outcome: Confident cost savings without SLO impact.

---

Common Mistakes, Anti-patterns, and Troubleshooting

List of mistakes with Symptom -> Root cause -> Fix (15–25 items):

1) Symptom: Gates failing frequently. -> Root cause: Overly strict rules or flaky checks. -> Fix: Triage failures, relax low-value rules, improve stability. 2) Symptom: High override rate. -> Root cause: Gates unusable or slow. -> Fix: Shorten checks, improve messaging, restrict overrides. 3) Symptom: Long pipeline delays. -> Root cause: Heavy simulations blocking CI. -> Fix: Move heavy checks to pre-merge or scheduled validations. 4) Symptom: Missing audit logs. -> Root cause: Not persisting check outputs. -> Fix: Centralize logs and attach artifacts to change ID. 5) Symptom: Post-deploy incidents despite pass. -> Root cause: Gaps between dry-run semantics and runtime. -> Fix: Add runtime simulation and canary linkage. 6) Symptom: False positives from drift. -> Root cause: Stale snapshots. -> Fix: Use live queries and refresh state prior to check. 7) Symptom: Flaky synthetic tests. -> Root cause: Environmental variability. -> Fix: Stabilize test harness and isolate dependencies. 8) Symptom: Cost predictions wildly off. -> Root cause: Inaccurate cost model. -> Fix: Improve model with historical usage and margins. 9) Symptom: Missing telemetry after deploy. -> Root cause: Instrumentation not validated. -> Fix: Enforce telemetry checks as required gate. 10) Symptom: Excessive policy violations. -> Root cause: Unmaintained rules. -> Fix: Regularly review and retire low-value policies. 11) Symptom: Developers bypass gates. -> Root cause: Poor UX or slow feedback. -> Fix: Improve feedback and integrate checks earlier. 12) Symptom: Admission controller conflicts with pipeline checks. -> Root cause: Duplicate enforcement with different rules. -> Fix: Harmonize policies across layers. 13) Symptom: Runbook steps outdated. -> Root cause: No validation of runbooks. -> Fix: Automate runbook testing and update cadence. 14) Symptom: High on-call interruptions for gate problems. -> Root cause: Alerts misrouted. -> Fix: Create clear routing for gate failures and secondary contacts. 15) Symptom: Over-reliance on autofix. -> Root cause: Blind trust in automation. -> Fix: Limit autofix to low-risk changes and require reviews for others. 16) Symptom: Checks block for network timeouts. -> Root cause: External dependency timeouts. -> Fix: Implement retries and circuit breakers. 17) Symptom: Policy engine slow under load. -> Root cause: Unoptimized ruleset. -> Fix: Profiling and caching of decision results. 18) Symptom: False negatives in dry-run. -> Root cause: Dry-run semantics differ from apply. -> Fix: Use provider-specific dry-run and integration tests. 19) Symptom: Metrics with high cardinality causing storage issues. -> Root cause: Per-change unique labels. -> Fix: Normalize labels and reduce cardinality. 20) Symptom: Teams disagree on gate ownership. -> Root cause: No clear operational model. -> Fix: Assign ownership and document responsibilities. 21) Symptom: Missing correlation between change and telemetry. -> Root cause: No changeID propagation. -> Fix: Propagate changeID across logs and traces. 22) Symptom: Alerts for non-critical policy changes. -> Root cause: No severity classification. -> Fix: Classify violations and route appropriately. 23) Symptom: Gate bypasses not audited. -> Root cause: Poor logging on overrides. -> Fix: Enforce logging and approval metadata.

Observability pitfalls (at least 5 included above):

• Missing audit logs, flaky synthetic tests, high cardinality metrics, missing changeID propagation, alerts misrouted.

---

Best Practices & Operating Model

Ownership and on-call:

• Platform team owns the pre-apply framework; product teams own check semantics for their scope.
• Dedicated on-call rotation for gate reliability.
• Clear escalation policy and secondary contacts.

Runbooks vs playbooks:

• Runbook: Operational step-by-step for known issues.
• Playbook: Higher-level decision trees for ambiguous situations.
• Keep runbooks runnable and tested; playbooks for cross-team decisions.

Safe deployments:

• Use canaries and gradual rollout after pre-apply checks pass.
• Predefine rollback triggers and automate rollbacks when SLOs violated.

Toil reduction and automation:

• Automate low-risk remediation and auto-verify.
• Reduce manual triage by surfacing clear remediation messages and links.

Security basics:

• Ensure audit logs are immutable and accessible to auditors.
• Encrypt decision artifacts and ensure least-privilege for gate components.
• Keep sensitive data out of logs.

Weekly/monthly routines:

• Weekly: Review gate failures and override events; prioritize flaky checks.
• Monthly: Policy review and owner confirmation; cost-model recalibration.
• Quarterly: Game day and runbook testing.

What to review in postmortems related to pre-apply checks:

• Whether the gate behaved as expected.
• Why the failed change reached production if gate passed.
• Override justification and whether it followed policy.
• Changes to checks, rules, or tooling required as outcome.

---

Tooling & Integration Map for pre-apply checks (TABLE REQUIRED)

ID	Category	What it does	Key integrations	Notes
I1	Policy engine	Evaluates policy-as-code rules	CI, admission, logging	Core gate for compliance
I2	Dry-run planner	Simulates infra apply plans	VCS, CI, cost tools	Provider semantics matter
I3	Metrics backend	Stores gate metrics	dashboards, alerts	Prometheus-style or managed
I4	Log aggregator	Stores decision logs and artifacts	audit, SRE tools	Centralized for postmortem
I5	Canary platform	Executes progressive rollouts	observability, traffic manager	Links pre-apply to runtime checks
I6	Synthetic runner	Runs user-like tests pre-deploy	CI, staging	Useful for realistic validation
I7	Cost estimator	Predicts infra cost delta	cloud billing, CI	Needs historical data
I8	Secrets manager	Validates secret presence and access	CI and runtime envs	Ensures credentials valid
I9	Runbook engine	Hosts runbooks and automated steps	incident systems, chatops	Automates remediation tasks
I10	GitOps operator	Applies approved manifests	VCS, policy engine	Enforces declarative flow

Row Details (only if needed)

• None

---

Frequently Asked Questions (FAQs)

What exactly is the difference between pre-apply and dry-run?

Dry-run is a simulation of apply; pre-apply is the gating stage that can include dry-run plus policy and telemetry checks.

Can pre-apply checks guarantee zero incidents?

No. They reduce risk but cannot guarantee zero incidents due to runtime uncertainties and external dependencies.

How long should a pre-apply check take?

Ideally under a few minutes; critical paths prefer <2 minutes. Longer checks should be scheduled or moved earlier in pipeline.

Should developers be able to bypass pre-apply checks?

Only via documented, auditable, and limited overrides with strict justification and approval.

Do pre-apply checks replace post-deploy monitoring?

No. They complement observability and canary analysis but cannot replace runtime monitoring.

How do you handle flaky pre-apply tests?

Triage and fix flakiness; mark flaky checks as advisory until stabilized; reduce false positives.

Are pre-apply checks necessary for small teams?

Depends. Start simple with linting and dry-run; scale complexity as risk and scale grow.

How to measure their effectiveness?

Use SLIs like gate pass rate, override rate, and post-deploy failure rate and monitor trends.

Can pre-apply checks be automated fully?

Many checks can be automated; cautious autofix must be limited and reversible.

How to manage policy rule sprawl?

Establish owners, periodic review cycles, and categorize rules by severity.

What if cost estimation is inaccurate?

Use conservative margins and historical usage to improve models; treat cost estimates as advisory if uncertain.

How do pre-apply checks fit into GitOps?

Checks can run on PRs and block merges; the GitOps operator applies only approved changes.

How to handle secrets in check logs?

Avoid writing secrets to logs and redact sensitive fields; use reference tokens.

What telemetry should every change require?

At minimum a health metric, error rate metric, and request latency for critical services.

How to scale pre-apply checks across many teams?

Provide a shared framework, reusable check templates, and self-service policy composer.

Can pre-apply checks use ML for anomaly detection?

Yes, but ML outputs should be advisory or combined with deterministic checks due to explainability concerns.

Who owns remediation of failed pre-apply checks?

Primary owner is the team that proposed the change; platform team supports gate infrastructure.

---

Conclusion

Pre-apply checks are a crucial safety net that reduces risk, preserves velocity, and enforces policy before changes reach production. When designed with speed, clarity, and observability, they prevent many common outages and provide auditable decision trails.

Next 7 days plan:

• Day 1: Inventory critical services and define required telemetry per service.
• Day 2: Add lightweight manifest linting and terraform plan validation in CI.
• Day 3: Integrate a simple policy-as-code check for one critical policy.
• Day 4: Export gate metrics to a monitoring backend and create basic dashboards.
• Day 5: Document runbooks for the top three gate failure modes and assign owners.

---

Appendix — pre-apply checks Keyword Cluster (SEO)

Primary keywords

• pre-apply checks
• pre apply checks
• pre-apply validation
• pre-deploy checks
• pre-deploy validation
• pre-apply gate
• pre-apply pipeline gate
• infra pre-apply checks

Secondary keywords

• infrastructure pre-apply
• policy-as-code gate
• CI pre-apply stage
• CI pipeline pre-apply
• terraform pre-apply
• kubernetes pre-apply checks
• serverless pre-apply validation
• dry-run pre-apply
• pre-apply audit
• pre-apply telemetry checks
• pre-apply canary integration

Long-tail questions

• what are pre-apply checks in CI/CD
• how to implement pre-apply checks for terraform
• why use pre-apply checks before deploying to production
• pre-apply checks vs dry-run vs admission controller
• best practices for pre-apply checks in kubernetes
• how to measure effectiveness of pre-apply checks
• how to prevent false positives in pre-apply checks
• what telemetry should pre-apply checks validate
• how long should pre-apply checks take
• how to automate pre-apply checks in CI
• can pre-apply checks include cost estimation
• how to audit pre-apply check decisions
• how to handle overrides for pre-apply checks
• pre-apply checks for database migrations
• integrating pre-apply checks with GitOps

Related terminology

• policy-as-code
• dry-run
• canary deployment
• admission controller
• GitOps
• terraform plan
• cost estimation
• synthetic tests
• observability validation
• telemetry coverage
• runbook automation
• decision engine
• override audit
• gate pass rate
• error budget
• SLI for gates
• compliance gate
• admission webhook
• mutation admission
• OPA Rego
• CI pipeline stage
• immutable infrastructure
• migration dry-run
• synthetic runner
• feature flag validation
• secrets validation
• IAM diff checker
• rollout strategy
• rollback plan
• sync loop validation
• architecture simulation
• agent-based validator
• policy violation rate
• change ID propagation
• audit trail completeness
• gate duration metric
• override policy
• automation autofix
• security scanner checklist
• telemetry linter
• observability completeness
• pre-apply checklist