I recently ran into a frustrating issue where Jenkins EC2 build agents were randomly disconnecting mid-build. What made it tricky to diagnose was that all memory and CPU metrics looked completely normal — no resource pressure, no obvious signs of stress on the agents.

Jenkins EC2 build agents were randomly going offline mid-build with the following errors:

AgentOfflineException: Unable to create live FilePath for EC2 (ci-cicd-agents) - build-ui-build (i-04ea4c67e83f0fea5);
EC2 (ci-cicd-agents) - build-ui-build (i-04ea4c67e83f0fea5) was marked offline: Connection was broken
...
ERROR: Issue with creating launcher for agent EC2 (ci-cicd-agents) - build-ui-build (i-04ea4c67e83f0fea5).
The agent has not been fully initialized yet
...
java.io.StreamCorruptedException: invalid stream header: 636F7272
  	at java.base/java.io.ObjectInputStream.readStreamHeader(Unknown Source)
  	at java.base/java.io.ObjectInputStream.<init>(Unknown Source)
  	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
  	at hudson.remoting.Command.readFrom(Command.java:141)
  	at hudson.remoting.Command.readFrom(Command.java:127)
  	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
  	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)

The controller logs showed the agent channel dying shortly after the build started:

INFO  hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel EC2 (ci-cicd-agents) - build-short-lived (i-0852a811f39adde70).
INFO  h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel EC2 (ci-cicd-agents) - build-short-lived (i-0852a811f39adde70)

The StreamCorruptedException with the header 636F7272 (corr in ASCII) hinted that the agent was sending garbled data over the remoting channel — not a network blip, but actual data corruption.

Root Cause Link to heading

JDK 17.0.14 had just been released. Our EC2 agents were running JDK 17.0.13 and had unattended-upgrades enabled as a systemd service.

The sequence of events:

  1. Jenkins provisions a new EC2 agent and connects to it via JVM remoting.
  2. At some point during the build, unattended-upgrades runs in the background and upgrades the JDK packages.
  3. The running JVM process detects or is affected by its own binaries being replaced on disk.
  4. The agent begins sending corrupt data over the open remoting channel to the controller, which manifests as the StreamCorruptedException.
  5. The controller marks the agent offline and the build fails.

Fix Link to heading

Disable unattended-upgrades on all Jenkins agents via Ansible:

- name: disable unattended-upgrades
  ansible.builtin.systemd_service:
    state: stopped
    enabled: false
    name: unattended-upgrades

Package upgrades on ephemeral CI agents should be handled at AMI bake time, not at runtime. Leaving unattended-upgrades enabled on short-lived agents is a footgun — any package that touches a running process can cause exactly this kind of silent, hard-to-reproduce failure.