Kubernetes CronJob Not Running: Diagnose Skipped Jobs and Monitor Completion

A warehouse ingestion pipeline stops producing fresh data, but kubectl get pods shows no failed containers. The expected Pod never existed. Kubernetes declined to create a new Job, so there is no application exception, container exit code, or failed Pod to investigate.

When diagnosing a Kubernetes CronJob not running, start above the Pod layer. Trace the execution chain:

CronJob schedule
    → Job creation
    → Pod creation
    → Pod scheduling and initialization
    → Application execution
    → Validated business result

Each transition can fail independently. Monitoring only the last few transitions leaves missing executions undetected.

1. The Phantom Pod: When Kubernetes Skips Your Workload

A CronJob is not a process waiting inside a container. It is a controller-managed resource that describes when Kubernetes should create Jobs.

The CronJob controller evaluates the schedule and relevant policy. The Job controller manages the resulting Pods. The scheduler places those Pods on nodes.

Consequently, an empty list of failed Pods tells you very little about whether yesterday’s database sync happened.

Several different conditions can produce stale data:

These are not necessarily silent inside Kubernetes. Events, status fields, and controller logs may explain the decision. They are silent to dashboards that look only for failed containers.

Inspect the CronJob before inspecting containers

Start with the resource and recent namespace activity:

NAMESPACE="batch"
CRONJOB="warehouse-sync"

kubectl -n "$NAMESPACE" get cronjob "$CRONJOB" -o yaml
kubectl -n "$NAMESPACE" describe cronjob "$CRONJOB"

kubectl -n "$NAMESPACE" get jobs \
  --sort-by=.metadata.creationTimestamp

kubectl -n "$NAMESPACE" get events \
  --field-selector "involvedObject.kind=CronJob,involvedObject.name=$CRONJOB" \
  --sort-by=.metadata.creationTimestamp

Check:

• .spec.schedule and .spec.timeZone
• .spec.suspend
• .spec.concurrencyPolicy
• .spec.startingDeadlineSeconds
• .status.active
• .status.lastScheduleTime
• .status.lastSuccessfulTime

Inspect Job owner references to distinguish executions belonging to this CronJob from unrelated Jobs.

If a Job exists, continue downward:

JOB="REPLACE_WITH_JOB_NAME"

kubectl -n "$NAMESPACE" describe job "$JOB"

kubectl -n "$NAMESPACE" get pods \
  -l "batch.kubernetes.io/job-name=$JOB" \
  -o wide

An absence of historical objects is not proof that execution never happened. Job history limits, TTL cleanup, and event retention can remove evidence. Preserve logs and execution metadata outside short-lived objects.

Collect the ownership chain without changing the cluster

Names and labels are convenient starting points, but a deleted CronJob can be recreated with the same name and a different UID. During an incident, join the actual controlling owner references before attributing old Jobs or unrelated Pods to the current CronJob.

Download the read-only CronJob evidence collector. It requires Node.js 18 or later and an installed, authenticated kubectl. Inspect the script before running it. Save it locally, then select your context, namespace and CronJob explicitly:

node kubernetes-cronjob-evidence.mjs \
  --context YOUR_CONTEXT \
  --namespace batch \
  --cronjob warehouse-sync > cronjob-evidence.json

The collector reads the CronJob, namespace Jobs, Pods and core Events, then reads the CronJob again to detect a changed identity. It matches the namespace, owner kind, API version, name, UID and controller: true. It reports observations along this chain:

Current CronJob UID
    → Jobs controlled by that CronJob
    → Pods controlled by each matching Job
    → Events matching the observed object identities

This uses only kubectl get operations. It does not create or restart a Job, modify a CronJob, read logs, request Secrets, contact PingCron or upload the report. Normal kubectl authentication can still invoke your configured authentication plugin. The account needs permission to get the named CronJob and list Jobs, Pods and core Events in that namespace.

Kubernetes returns objects to kubectl, and the script processes their JSON locally before selecting fields for the report. The saved output omits workload commands, environment values, full manifests, arbitrary annotations and event messages. It retains resource names, UIDs, timing, scheduling policy, status and reasons. Review those identifiers before sharing it; this is not a complete privacy scrub of every field an operator could populate.

Exit code 0 means the collection completed, not that the workload succeeded. Exit code 2 means invalid arguments or incomplete collection. API errors, timeouts, malformed responses, unfinished pagination and a CronJob replaced during collection cannot silently become an empty successful report. The script suppresses raw command errors because authentication plugins and server errors can include sensitive values; investigate the corresponding read directly with kubectl when needed.

The reads have separate timestamps and are not an atomic snapshot. Jobs can appear or disappear between calls. The collector limits each request and the amount of JSON it accepts; a namespace that exceeds those limits produces an incomplete report rather than a partial success.

Ownership does not prove that the scheduler ran. kubectl create job --from=cronjob/... can set the same CronJob owner reference as a scheduled Job. The collector labels the cronjob.kubernetes.io/instantiate: manual annotation when present. Missing that annotation means the origin is not established, not automatically scheduled. It also shows the scheduled-timestamp annotation when supplied by the controller; Kubernetes added that annotation in version 1.32. Compare the annotation and retained history with the expected occurrence, rather than counting every owned Job as a schedule success. kubectl manual Job construction · CronJob Job-creation documentation.

The collector is an evidence aid, not a root-cause classifier. Its 14 fixture tests cover identity reuse, misleading labels, manual Jobs, incomplete reads, retained-object gaps and retries. Fixture tests verify report behavior; they do not reproduce every cluster version, controller or admission policy.

We also ran the collector against a disposable Kubernetes v1.34.11 cluster on November 11, 2026. Download the captured runtime observations, including versions, synthetic manifests and per-case reports. A scheduled Job completed; a suspended CronJob retained no Jobs during the 74.67-second observation window; an invalid image produced ImagePullBackOff; and a process exiting 23 produced a failed Job. A manual Job created from the suspended CronJob still appeared in its ownership chain with the manual annotation. The fixture cluster was deleted and its absence verified. These are synthetic cases on one Kubernetes version, not production incidents or proof of compatibility with every cluster.

2. The Architecture Core: Tuning concurrencyPolicy and startingDeadlineSeconds

Two settings govern different decisions:

• concurrencyPolicy determines how this CronJob handles overlapping Jobs.
• startingDeadlineSeconds determines how late an occurrence remains eligible for Job creation.

Neither setting proves successful application completion.

concurrencyPolicy: Forbid can repeatedly block new work

Consider this configuration fragment:

spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 120

Suppose a Job starts at 12:00 and remains active until 12:18:

This is deliberate policy enforcement. It does not require a failed container.

An active Job can be doing useful work, blocked on a database lock, waiting on a dependency, or unable to complete its Pods. Without a runtime bound, it can obstruct subsequent executions for an extended period.

The controller can emit a JobAlreadyActive event when Forbid prevents creation. Inspect normal events as well as warnings. Kubernetes CronJob controller implementation

The alternatives have different consequences:

These policies apply to Jobs from the same CronJob. They are not a distributed lock across other CronJobs or independently created Jobs. CronJob API reference

For workloads with side effects, enforce idempotency or application-level locking where required.

startingDeadlineSeconds bounds late Job creation

This configuration is fragile:

spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 1

It gives the controller approximately one second after an expected occurrence to create its Job.

Kubernetes explicitly warns that values below ten seconds may prevent scheduling. A deadline that small leaves almost no tolerance for controller delays or control-plane disruption. Kubernetes CronJob limitations

However, being one second late is not universally fatal. The configured deadline determines the tolerance.

For example:

spec:
  schedule: "17 2 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600

This permits late Job creation within a ten-minute window.

It does not require the Pod to start executing within ten minutes. If Kubernetes creates the Job promptly but its Pod waits for capacity, image pulls, or storage, that is a later stage.

Use a Job execution deadline separately:

spec:
  jobTemplate:
    spec:
      activeDeadlineSeconds: 1800
      backoffLimit: 0

activeDeadlineSeconds bounds the Job’s active duration. When the deadline is enforced, Kubernetes terminates running Pods and marks the Job failed. It takes precedence over further retries. Kubernetes Job termination documentation

More than 100 missed schedules does not permanently stop the controller

There are two separate issues that should not be combined into one failure story:

1. A short starting deadline can make an individual occurrence ineligible.
2. A long catch-up window can contain many missed occurrences.

A one-second deadline on a once-per-minute schedule cannot itself contain more than 100 missed executions.

The Kubernetes concept documentation describes a catch-up limit when more than 100 schedules were missed. It also explicitly states that this does not mean the CronJob stops running permanently. CronJob catch-up documentation

There is an additional version-sensitive detail: the Kubernetes v1.34.0 implementation emits a TooManyMissedTimes warning and returns the most recent eligible schedule. The threshold itself does not unconditionally abort scheduling in that implementation. Versioned controller source

Therefore, do not diagnose a permanent scheduling shutdown from that warning alone.

Inspect your cluster version, current eligibility window, active Jobs, and events. The relevant missed-schedule window is bounded by the starting deadline when one is configured; otherwise, older scheduling history can contribute.

A useful operational approximation for a regular schedule is:

Missed occurrences ≈ relevant look-back duration / schedule interval

A daily schedule and a once-per-minute schedule accumulate missed occurrences at very different rates.

Choose the starting deadline according to how late the business operation may safely begin. Do not shrink it solely to suppress a warning. CronJob catch-up is not a durable queue that guarantees replay of every missed business batch.

3. The Monitoring Blindspot: Why Pod Metrics Are Not Enough

A Pod that was never created cannot:

• Exit with a nonzero code.
• Become OOMKilled.
• Emit application logs.
• Increment its container restart count.

An alert limited to those signals cannot detect an absent execution.

But Prometheus is not inherently incapable of detecting missing CronJobs. That would be an incorrect distinction.

kube-state-metrics exposes CronJob metrics including:

• kube_cronjob_status_last_schedule_time
• kube_cronjob_status_last_successful_time
• kube_cronjob_status_active
• kube_cronjob_spec_suspend

These support explicit overdue-execution and overdue-success rules. Official CronJob metric reference

Prometheus also supports absent() and absent_over_time() for detecting missing time series. Prometheus query functions

The operational limitations are configuration and failure independence:

• A last-success age rule needs handling for jobs that have never succeeded.
• Missing metrics must not silently turn an alert expression into an empty result.
• Kubernetes Job success does not establish data correctness.
• Prometheus and Alertmanager running only inside the affected cluster share some of its failure modes.

Use cluster metrics for diagnosis and local detection. Add an external completion expectation when the cluster itself must not be the sole place capable of noticing missing work.

4. Implementing a Secure Heartbeat Loop Inside the Pod Spec

The heartbeat must follow completed and validated work.

Two ordinary containers listed under containers: do not execute sequentially. A separate curl container can send success while the application is still running.

The following design makes ordering explicit:

Ordinary init container:
    execute sync
    verify destination state
    exit zero
        |
        v
Reporter container:
    send success heartbeat
    exit

Ordinary init containers must complete successfully before application containers start. With restartPolicy: Never, a failed init container causes the Pod to fail. Kubernetes init-container behavior

This is one practical way to keep the monitoring credential out of the workload container.

Provision the Secret

Create the namespace through your normal infrastructure process. Store the monitor’s private success URL in a protected file, then create the Secret without putting the URL directly in the command line:

kubectl -n batch create secret generic warehouse-sync-pingcron \
  --from-file=url=/secure/path/pingcron-url

Use your secret manager or encrypted GitOps workflow for ongoing management.

Kubernetes Secrets are not automatically confidential merely because their values are base64 encoded. Restrict access, enable encryption at rest, and do not commit plaintext Secret manifests. Kubernetes Secret security guidance

Complete CronJob manifest

This template requires an application image containing:

• /bin/sh
• python3
• /app/sync.py
• /app/verify_sync.py

The first script must wait for the sync to finish. The second must verify the required destination state. Both must return nonzero on failure.

Replace the example application image and add its workload-specific credentials. The image tags below make the template readable; resolve both images to reviewed immutable digests before production deployment.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: warehouse-sync
  namespace: batch
spec:
  schedule: "17 2 * * *"
  timeZone: "Etc/UTC"
  suspend: false

  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600

  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5

  jobTemplate:
    metadata:
      labels:
        app.kubernetes.io/name: warehouse-sync
    spec:
      completions: 1
      parallelism: 1

      # Bound Job duration, including initialization and reporting.
      activeDeadlineSeconds: 1800

      # Avoid automatically repeating completed mutations if reporting fails.
      backoffLimit: 0

      template:
        metadata:
          labels:
            app.kubernetes.io/name: warehouse-sync
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          terminationGracePeriodSeconds: 30

          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            runAsGroup: 10001
            fsGroup: 10001
            seccompProfile:
              type: RuntimeDefault

          initContainers:
            - name: sync-and-validate
              image: registry.example.com/platform/warehouse-sync:2026-10-31
              imagePullPolicy: IfNotPresent

              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -u

                  python3 /app/sync.py
                  sync_status=$?

                  if [ "$sync_status" -ne 0 ]; then
                    printf 'Sync failed with exit code %s\n' \
                      "$sync_status" >&2
                    exit "$sync_status"
                  fi

                  python3 /app/verify_sync.py
                  verify_status=$?

                  if [ "$verify_status" -ne 0 ]; then
                    printf 'Validation failed with exit code %s\n' \
                      "$verify_status" >&2
                    exit "$verify_status"
                  fi

                  printf '%s\n' 'Sync and validation completed.'
                  exit 0

              env:
                - name: PYTHONDONTWRITEBYTECODE
                  value: "1"
                - name: PYTHONUNBUFFERED
                  value: "1"

              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]

              resources:
                requests:
                  cpu: "250m"
                  memory: "256Mi"
                limits:
                  cpu: "1"
                  memory: "1Gi"

              volumeMounts:
                - name: scratch
                  mountPath: /tmp

          containers:
            - name: report-success
              image: python:3.13-slim
              imagePullPolicy: IfNotPresent

              command: ["python3", "-u", "-c"]
              args:
                - |
                  import os
                  import sys
                  import time
                  import urllib.error
                  import urllib.parse
                  import urllib.request

                  url = os.environ["PINGCRON_URL"].strip()
                  parsed = urllib.parse.urlsplit(url)

                  if parsed.scheme != "https" or not parsed.hostname:
                      print("Invalid heartbeat URL configuration.",
                            file=sys.stderr)
                      sys.exit(1)

                  class NoRedirect(urllib.request.HTTPRedirectHandler):
                      def redirect_request(self, req, fp, code, msg,
                                           headers, newurl):
                          return None

                  opener = urllib.request.build_opener(NoRedirect())

                  for attempt in range(3):
                      request = urllib.request.Request(
                          url,
                          data=b"",
                          method="POST",
                      )

                      try:
                          with opener.open(request, timeout=10) as response:
                              if 200 <= response.status < 300:
                                  print("Completion heartbeat accepted.")
                                  sys.exit(0)

                      except (urllib.error.URLError, TimeoutError):
                          # Do not print exceptions containing the secret URL.
                          pass

                      if attempt < 2:
                          time.sleep(5)

                  print(
                      "Workload completed, but heartbeat delivery failed.",
                      file=sys.stderr,
                  )
                  sys.exit(1)

              env:
                - name: PINGCRON_URL
                  valueFrom:
                    secretKeyRef:
                      name: warehouse-sync-pingcron
                      key: url
                      optional: false
                - name: PYTHONDONTWRITEBYTECODE
                  value: "1"

              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]

              resources:
                requests:
                  cpu: "25m"
                  memory: "32Mi"
                limits:
                  cpu: "100m"
                  memory: "128Mi"

          volumes:
            - name: scratch
              emptyDir:
                sizeLimit: 1Gi

Why this ordering matters

The reporter cannot start unless the workload container exits successfully after both commands.

The application container does not receive PINGCRON_URL. It also does not receive a Kubernetes service-account token by default.

The reporter uses HTTPS verification, rejects redirects, limits individual HTTP operations, retries only reporting, and avoids logging the secret URL. The Job deadline provides the broader execution bound.

Configure network policy to allow the required database, storage, DNS, and HTTPS traffic. Standard Kubernetes NetworkPolicy does not provide portable hostname-based egress rules; use your CNI’s supported mechanism or an approved egress proxy where necessary.

Interpret Job failure carefully

A reporting failure occurs after the sync has already committed its changes. This template marks the Job failed but uses backoffLimit: 0 to avoid automatic failure retries.

That is not an exactly-once guarantee. Workloads must still tolerate duplicate execution, recovery, and deliberate reruns. Inspect the failed container before repeating a mutation.

Because the workload runs as an init container, retrieve its logs explicitly:

kubectl -n batch logs "job/$JOB" -c sync-and-validate
kubectl -n batch logs "job/$JOB" -c report-success

Finally, a heartbeat confirms what the workload reported. The validation script must check meaningful invariants, such as the destination watermark, required partition, reconciliation totals, or freshness of the current batch.

5. Aligning Cluster Grace Periods With Platform Realities

Create a companion PingCron monitor with the same schedule and timezone:

PingCron watches for the expected success receipt and can alert when it remains absent beyond grace. Verify the receipt and notification destination before relying on the monitor. PingCron’s monitoring model

Budget for the entire path to completion

The grace period starts from the expected occurrence, not from the moment the application begins executing.

Account for:

Grace period >=
    controller delay before Job creation
  + Pod admission and scheduling
  + node provisioning
  + image pulls and initialization
  + application execution
  + output validation
  + heartbeat delivery
  + operating margin

For the manifest above, a useful budget is:

Allowed late Job creation:       10 minutes
Job active-duration budget:      30 minutes
Additional operating margin:      5 minutes
Monitor grace:                   45 minutes

The 30-minute Job budget must accommodate provisioning, image pulls, work, validation, and reporting after the Job becomes active. Do not add those components again when calculating from that aggregate budget.

Measure actual behavior under node scaling and cold image pulls. If your workload legitimately needs more time, adjust both the Job deadline and monitor grace.

If the business deadline is stricter than the platform’s normal completion time, increasing grace does not solve the scheduling problem.

Verify successful execution first

Before creating workloads:

kubectl apply --dry-run=server -f warehouse-sync.yaml

Server-side validation checks the manifest against the cluster API. It does not prove that the image, credentials, network paths, or application will work.

In a staging environment, you can create a manual Job from the CronJob template:

kubectl -n batch-test create job \
  warehouse-sync-manual-test \
  --from=cronjob/warehouse-sync-test

Use a unique name for subsequent tests.

Verify the actual destination data, workload logs, reporter response, and PingCron receipt. Then observe a real scheduled execution separately.

A manually created Job does not validate CronJob scheduling and should not be used to infer that Forbid has protected it from overlap.

Test a missing run by suspending an isolated CronJob

Create a test CronJob and separate monitor using harmless fixture data. A short schedule makes the test practical.

Before suspension, ensure no active test Job remains that could send a late heartbeat.

Suspend only the test CronJob:

kubectl -n batch-test patch cronjob warehouse-sync-test \
  --type=merge \
  -p '{"spec":{"suspend":true}}'

Keep the PingCron monitor enabled.

CronJob suspension stops future scheduled executions; it does not stop Jobs already created. This differs from suspending an individual Job. CronJob suspension semantics

After the next expected occurrence and grace period:

1. Confirm no new scheduled test Job was created.
2. Confirm the expected heartbeat is absent.
3. Confirm the external Slack, Discord, or Email notification arrived.
4. Verify that the notification identifies the correct monitor.

If GitOps manages the object, make the test change through that process or account for reconciliation restoring the previous value.

Restore scheduling:

kubectl -n batch-test patch cronjob warehouse-sync-test \
  --type=merge \
  -p '{"spec":{"suspend":false}}'

Resuming can make a recent missed occurrence eligible for catch-up, depending on the deadline. Keep the test workload safe to rerun.

Also test application failure separately by making the isolated sync or validation command exit nonzero. The reporter must remain unstarted.

A missing heartbeat does not identify the root cause. Return to the execution chain to distinguish a skipped occurrence, pending Pod, failed application, failed validation, and reporting failure.

6. Conclusion: Keep the Completion Check Outside the Failure Domain

A healthy cluster dashboard does not establish that a scheduled business operation completed. An absent Pod cannot emit a container failure, and a successful process can still leave stale data.

Use Kubernetes events, Job status, and explicit Prometheus rules to diagnose execution. Maintain a separate completion expectation when the cluster cannot be the only system responsible for reporting its own failure.

This matters even more on a single-node cluster. If workloads, metrics, and alert delivery all depend on that node, its loss can remove both the work and the warning system.

Start with one high-value task. Register a free PingCron monitor, report success only after validated completion, and test a suspended non-production CronJob before relying on the external alert.

Related monitoring guides

For configuration and setup, see the Kubernetes CronJob validator.

• How to Monitor Postgres Backup Cron Jobs
• GitHub Actions Scheduled Workflow Not Running?
• Laravel Scheduled Job Not Running? Monitor Workers
• Self Hosted vs Hosted Cron Monitoring: Tool Comparison