Fix Jenkins EC2 Agent Disconnecting Mid-Build

Tue, 06 Feb 2024 00:00:00 +0000

I recently ran into a frustrating issue where Jenkins EC2 build agents were randomly disconnecting mid-build. What made it tricky to diagnose was that all memory and CPU metrics looked completely normal — no resource pressure, no obvious signs of stress on the agents.

Jenkins EC2 build agents were randomly going offline mid-build with the following errors:

AgentOfflineException: Unable to create live FilePath for EC2 (ci-cicd-agents) - build-ui-build (i-04ea4c67e83f0fea5);
EC2 (ci-cicd-agents) - build-ui-build (i-04ea4c67e83f0fea5) was marked offline: Connection was broken
...
ERROR: Issue with creating launcher for agent EC2 (ci-cicd-agents) - build-ui-build (i-04ea4c67e83f0fea5).
The agent has not been fully initialized yet
...
java.io.StreamCorruptedException: invalid stream header: 636F7272
 	at java.base/java.io.ObjectInputStream.readStreamHeader(Unknown Source)
 	at java.base/java.io.ObjectInputStream.<init>(Unknown Source)
 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
 	at hudson.remoting.Command.readFrom(Command.java:141)
 	at hudson.remoting.Command.readFrom(Command.java:127)
 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)

The controller logs showed the agent channel dying shortly after the build started:

Ec2 on Adam Vu

Fix Jenkins EC2 Agent Disconnecting Mid-Build