Member-only story
How intuition makes us better engineers
Becoming a Better Engineer — Part 1
The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honors the servant and has forgotten the gift. — Albert Einstein
How it all started
The realization that intuition was going to be important happened approximately 15 years ago.
Cloud computing was still in its infancy and the company I worked with was one of the early adopters. We had a simple 3-tier application comprised of a load-balancer, the backend application, and a database.
At the time, AWS didn’t offer any managed load balancer so we had to deploy our own ones.
Thanks to a successful sale, our traffic was going to triple almost over night.
Shortly after the traffic started to increase, alarms set off. Client errors went through the roof and the load traffic graph looked something like that.
Notice the flat line?
Back then, I had no idea what was going on. Since I was the one responsible for the system’s operation, I started to panic.
My boss looked at the graph and like it was trivial, said something like:
“It sure looks like a limit issue. The load balancer’s configuration are probably set to the default values.”
And he was right. limit.conf and nginx.conf were the guilty ones, set to their default values.
While it was obvious to him, it wasn’t to me.
Afterwards, I asked him how he knew.
“I didn’t. It was a hunch.”
My boss had 20 years more experience than me, and it turns out he had seen something similar in the past and quickly concluded the graph pattern was matching something that he felt was related to limits.
