{"id":296,"date":"2026-09-11T11:29:57","date_gmt":"2026-09-11T11:29:57","guid":{"rendered":"https:\/\/www.codelogicx.com\/blog\/?p=296"},"modified":"2026-09-11T11:55:30","modified_gmt":"2026-09-11T11:55:30","slug":"aws-alb-debugging","status":"publish","type":"post","link":"https:\/\/www.codelogicx.com\/blog\/aws-alb-debugging\/","title":{"rendered":"Two-Line Fix Took Three Days: How We Debugged AWS ALB Issues"},"content":{"rendered":"

This blog aims to share the lessons learned from a deceptively simple security remediation task that turned into a three-day investigation across multiple layers of cloud infrastructure. Through the journey of disabling port 80 on a shared Application Load Balancer (ALB), uncovering a silent IAM role fallback, and identifying an overlooked health check dependency, readers will gain practical insights into troubleshooting complex AWS environments. <\/span>\u00a0<\/span><\/p>\n

This content\u00a0emphasizes the importance of understanding shared architecture, validating assumptions against actual deployments, and recognizing how small configuration changes can create unexpected downstream impacts. <\/span>\u00a0<\/span><\/p>\n

Ultimately, it serves as a reminder that effective debugging is often less about fixing code and more about uncovering hidden relationships within modern cloud systems.\u00a0<\/span>\u00a0<\/span><\/p>\n

Let’s dive deep into\u00a0the steps\u00a0how two low-severity security issues exposed:\u00a0\u00a0\u00a0<\/span>\u00a0<\/span><\/p>\n

The Issue flagged during Penetration Testing:<\/span>\u00a0<\/span><\/h2>\n

It arrived as one of the least threatening items on the board. A penetration test had flagged two findings:<\/span>\u00a0<\/span><\/p>\n

Insecure Ports in Use.<\/span><\/b> Endpoints are reachable over plain HTTP on port 80.Requests are 301-redirected to HTTPS, but the initial connection is unencrypted.<\/span>\r\n\r\nServer Version Disclosure.<\/span><\/b> HTTP responses expose <\/span>Server:awselb\/2.0<\/span>,disclosing the load balancer technology.<\/span><\/pre>\n
Two fixes. One removes a listener. One sets a boolean. Both are documented by AWS in a single paragraph each. I estimated an afternoon.<\/span>\u00a0<\/span><\/p>\n
This is what happened.<\/span>\u00a0<\/span><\/p>\n
At first glance, this looked like routine security housekeeping.<\/span>\u00a0<\/span><\/p>\n
The remediation plan was straightforward. One finding could be addressed by removing a listener. The other required flipping a boolean configuration value. AWS documentation covered both changes in a few short paragraphs. Nothing about the work suggested risk, complexity, or even much effort.<\/span>\u00a0<\/span><\/p>\n
I estimated an afternoon, but\u00a0the\u00a0entire task consumed three days.<\/span>\u00a0<\/span><\/p>\n
What began as two minor hardening changes led to the discovery of a production IAM misconfiguration that had been quietly active for 93 days. Along the way, a seemingly unrelated CloudWatch alarm started firing and triggered another investigation. That alarm, as it turned out, was tied to a health check that nobody had ever actually validated.<\/span>\u00a0<\/span><\/p>\n
By the end of it, the penetration test findings were the easiest part of the entire exercise.<\/span>\u00a0<\/span><\/p>\n
This is the story of how two low-severity security issues exposed a chain of assumptions hiding in plain sight, and why some of the simplest infrastructure changes can reveal the most interesting problems.<\/span>\u00a0<\/span><\/p>\n
In this blog, we have segregated the blog in\u00a0<\/span>Different Acts\u00a0for better understanding.\u00a0<\/span><\/i><\/b>\u00a0<\/span><\/p>\n
Act One: The Shared Load Balancer<\/span>\u00a0<\/span><\/h2>\n
The first fix looked trivial. In the Kubernetes Ingress manifest:\u00a0<\/strong><\/p>\n
# Before<\/span>\r\nalb.ingress.kubernetes.io\/listen-ports: '[{\"HTTP\": 80}, {\"HTTPS\":443}]'<\/span>\r\nalb.ingress.kubernetes.io\/ssl-redirect: '443'<\/span>\r\n# After<\/span> \r\nalb.ingress.kubernetes.io\/listen-ports: '[{\"HTTPS\":443}]'<\/span><\/pre>\n
Delete the redirect, drop the listener. Deploy. Done.<\/span>\u00a0<\/span><\/p>\n
Except port 80 stayed open.<\/span>\u00a0<\/span><\/p>\n
The reason was a single annotation I’d skimmed past:<\/span>\u00a0<\/span><\/p>\n
alb.ingress.kubernetes.io\/group.name: ingress-backend-group<\/span><\/pre>\n
The AWS Load Balancer Controller supports <\/span>ingress groups. <\/span><\/b>These<\/span> multiple Ingress resources\u00a0that are potentially across different namespaces and different Git repositories, merge into a single physical ALB. It’s an excellent cost optimization. It’s also a footgun if you don’t know it’s there.<\/span>\u00a0<\/span><\/p>\n
The controller computes the union of every group member’s requirement. If <\/span>any<\/span><\/i> ingress in the group asks for port 80, the listener exists\u00a0for everyone.<\/span>\u00a0<\/span><\/p>\n
\"\"<\/p>\n
I had fixed one repository. There were four more and\u00a0one ingress that existed in no repository at all, applied years earlier with <\/span>kubectl apply<\/span> and never captured in Git.<\/span>\u00a0<\/span><\/p>\n
The lesson here is not “read your annotations.” It’s that <\/span>infrastructure boundaries and repository boundaries are different things<\/span><\/b>, and nothing in a code review will tell you that the file you’re editing shares state with four repos you’ve never opened.<\/span>\u00a0<\/span><\/p>\n
Finding them required stepping outside Git entirely and asking the cluster:<\/span>\u00a0<\/span><\/p>\n
kubectl get ingress -A -o json | jq -r '<\/span>\r\n\u00a0 .items[] |<\/span>\r\n\u00a0 select(.metadata.annotations[\"alb.ingress.kubernetes.io\/group.name\"] == \"ingress-backend-group\") |<\/span>\r\n\u00a0 \"\\(.metadata.name)\\t\\(.metadata.annotations[\"alb.ingress.kubernetes.io\/listen-ports\"])\"<\/span><\/pre>\n
\u00a0<\/span><\/p>\n
One query, complete answer. I should have run it on\u00a0the first day.<\/span>\u00a0<\/span><\/p>\n
Act Two: The Silent Fallback<\/span>\u00a0<\/span><\/p>\n
With every manifest corrected and deployed, <\/span>port 80<\/span><\/b> still would not close.<\/span>\u00a0<\/span><\/p>\n
The controller logs explained why, and the explanation was more interesting than the problem:<\/span>\u00a0<\/span><\/p>\n
UnauthorizedOperation: You are not authorized to perform this operation.<\/span>\r\nUser: arn:aws:sts::REDACTED:assumed-role\/eksctl-...-NodeInstanceRole-...\/i-...<\/span>\r\nis not authorized to perform: ec2:RevokeSecurityGroupIngress<\/span><\/pre>\n
\u00a0<\/span><\/p>\n
Removing a listener also means revoking the corresponding security group rule. The controller tried, but AWS refused.<\/span>\u00a0<\/span><\/p>\n
The interesting part is the principal. The controller was making that call as the <\/span>EC2 node instance role<\/span><\/b>\u00a0the generic identity attached to the worker node\u00a0not as its own dedicated IAM role.<\/span>\u00a0<\/span><\/p>\n
This is the failure mode of IRSA (IAM Roles for Service Accounts). When IRSA is wired correctly, a webhook injects credentials into the pod at creation time:<\/span>\u00a0<\/span><\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
The Critical Detailing: <\/span>\u00a0<\/span><\/h3>\n
There is no error when this fails.<\/span><\/b> The AWS SDK credential chain is designed to fall through gracefully. IRSA credentials absent? Try environment variables. Absent? Try instance metadata. Something always answers. The pod starts, reports healthy, and runs indefinitely with the wrong identity\u00a0until it attempts an operation the fallback role can’t perform.\u00a0<\/span>\u00a0<\/span><\/p>\n
In this case, that\u00a0took 93 days.<\/span>\u00a0<\/span><\/p>\n
The diagnosis includes three checks:\u00a0<\/strong><\/p>\n
# 1. Is the annotation on the ServiceAccount?\u00a0 (It was.)<\/span>\r\nkubectl get sa aws-load-balancer-controller -n kube-system \\<\/span>\r\n\u00a0 -o jsonpath='{.metadata.annotations.eks\\.amazonaws\\.com\/role-arn}'<\/span>\r\n# 2. Does the IAM role have the permission?\u00a0 (It did.)<\/span>\r\naws iam get-policy-version --policy-arn <policy> --version-id <v><\/span>\r\n# 3. Does the running pod actually have IRSA credentials?\u00a0 (It did not.)<\/span>\r\nkubectl get pod <controller-pod> -n kube-system \\<\/span>\r\n\u00a0 -o jsonpath='{.spec.containers[0].env[*].name}' | tr ' ' '\\n' | grep AWS<\/span>\r\n# \u2192 empty<\/span><\/pre>\n
Configurations\u00a0are correct. Permissions correct. Running process wrong. The annotation had been added <\/span>after<\/span><\/i> the pod was created, and nothing had restarted it since.<\/span>\u00a0<\/span><\/p>\n
Fixing it without a gap<\/span>\u00a0<\/span><\/h2>\n
The controller was running a single replica. A naive <\/span>kubectl rollout restart<\/span> would leave a window with no controller at all. Instead:<\/span>\u00a0<\/span><\/p>\n
\"\"<\/p>\n
The explicit stop-gate matters. If the new pod comes up <\/span>also<\/span><\/i> lacking the token volume, the webhook isn’t firing and restarting again will not help. All\u00a0you\u00a0would\u00a0just be cycling pods while the real problem sits elsewhere. Encoding “if this check fails, stop” into the runbook prevents the instinct to try again harder.<\/span>\u00a0<\/span><\/p>\n
The restart worked. The controller assumed its correct role, reconciled, removed the listener, revoked the security group rule. It also silently fixed a second permission error nobody had reported\u00a0the same fallback had been blocking AWS Shield subscription checks for the entire 93 days.\u00a0<\/span><\/p>\n
Act Three: Two Documentation Pages, One Truth<\/span>\u00a0<\/span><\/h2>\n
The second finding is- suppressing Server: awselb\/2.0 \u2014 had a documented one-line fix:\u00a0<\/strong><\/p>\n
routing.http.response.server.enabled = false<\/span><\/pre>\n
The question was where to put it. And this is where things got genuinely confusing, because <\/span>AWS documents this attribute in two different places with two different APIs.<\/span><\/b>\u00a0<\/span><\/p>\n
One page shows it set via <\/span>modify-load-balancer-attributes<\/span>. Another shows it under listener attributes, set via <\/span>modify-listener-attributes<\/span>. Both are official AWS documentation.<\/span>\u00a0<\/span><\/p>\n
The first attempt placed it as a load-balancer attribute. AWS rejected it:\u00a0<\/strong><\/p>\n
ValidationError: Load balancer attribute key<\/span>\r\n'routing.http.response.server.enabled' is not recognized<\/span><\/pre>\n
Worse, because the value was embedded in an Ingress annotation, the controller retried the rejected call on <\/span>every reconciliation loop<\/span><\/b> \u2014 and a failing reconcile blocks <\/span>all<\/span><\/i> other pending changes for that ingress group. The invalid attribute for finding \u00a73.6 was actively preventing the fix for \u00a73.5 from applying.<\/span>\u00a0<\/span><\/p>\n
Two findings, tangled together by a shared reconciliation loop.<\/span>\u00a0<\/span><\/p>\n
The resolution was to stop reading documentation and ask the environment directly:<\/span>\u00a0<\/span><\/p>\n
aws elbv2 describe-listener-attributes --listener-arn <443-listener><\/span><\/pre>\n
{<\/span>\r\n\u00a0 \"Key\": \"routing.http.response.server.enabled\",<\/span>\r\n\u00a0 \"Value\": \"true\"<\/span>\r\n}<\/span><\/pre>\n
There it was. A <\/span>listener<\/span><\/b> attribute, on this ALB, in this region, right now. No ambiguity, no interpretation.<\/span>\u00a0<\/span><\/p>\n
When documentation conflicts, the environment is the source of truth.<\/span><\/b> A <\/span>describe-*<\/span> call takes two seconds and settles arguments that documentation, blog posts, and automated code review cannot.<\/span><\/pre>\n
The controller version constraint<\/span>\u00a0<\/span><\/h2>\n
Knowing the correct placement didn’t mean it could be managed declaratively. The installed controller version predated support for the <\/span>listener-attributes<\/span> annotation \u2014 the correct annotation would simply be ignored. <\/span>\u00a0<\/span><\/p>\n
So, three options existed:\u00a0<\/em><\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t
Approach <\/th>Works today<\/th>GitOps-managed<\/th>Self-healing<\/th>\n<\/tr>\n<\/thead>\n
Load-balancer attribute annotation<\/td>No \u2014 API rejects it<\/td><\/td><\/td>\n<\/tr>\n
Listener attribute annotation<\/td>No \u2014 controller too old<\/td>Yes<\/td>Yes<\/td>\n<\/tr>\n
Direct CLI \/ console on listener<\/td>Yes<\/td>No<\/td>No<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n
The pragmatic choice was the third, with an explicit follow-up to upgrade the controller and migrate to the declarative path.<\/span>\u00a0<\/span><\/p>\n
But applying configuration outside GitOps to a controller-managed resource raises an obvious question: <\/span>will the controller revert it?<\/span><\/b>\u00a0<\/span><\/p>\n
Rather than assume, I tested it. A throwaway annotation forces a full reconcile:\u00a0<\/strong><\/p>\n
kubectl annotate ingress <name> reconcile-test=\"$(date +%s)\" --overwrite<\/span>\r\nsleep 90<\/span>\r\n# re-check the attribute<\/span>\r\nkubectl annotate ingress <name> reconcile-test-<\/span><\/pre>\n
\u00a0<\/span><\/p>\n
The value survived. Because the controller version doesn’t manage listener attributes at all, it leaves them untouched\u00a0the same limitation that prevented the declarative fix also guaranteed the manual one would persist.<\/span>\u00a0<\/span><\/p>\n
That’s a satisfying symmetry, but the important part is that it was <\/span>verified rather than assumed<\/span><\/b>. “It’ll probably be fine” is not a deployment strategy.<\/span>\u00a0<\/span><\/p>\n
Act Four: The Alarm That Wasn’t Our Fault (But Looked Like It)<\/span>\u00a0<\/span><\/h2>\n
Days later, a CloudWatch alarm fired:\u00a0<\/strong><\/p>\n
Production-API-NoHealthyTargets<\/span>\r\nHealthyHostCount < 1 for 2 datapoints<\/span><\/pre>\n
Zero healthy targets on a production API target group \u2014 and it belonged to one of the exact services I’d modified.<\/span>\u00a0<\/span><\/p>\n
The obvious conclusion was that my change had broken it. The obvious conclusion was wrong, and the process of establishing that is worth more than the fix.<\/span>\u00a0<\/span><\/p>\n
What the evidence showed:<\/h2>\n
Pods were <\/span>1\/1 Running<\/span>, 43 hours old, no restarts. The Kubernetes liveness probe was passing. But the ALB reported every target as:<\/span>\u00a0<\/span><\/p>\n
Target.ResponseCodeMismatch: Health checks failed with these codes: [302]<\/span><\/pre>\n
Curling the application directly from inside the pod:\u00a0<\/strong><\/p>\n
$ curl -sI http:\/\/localhost:3005\/<\/span>\r\nHTTP\/1.1 302 Found<\/span>\r\nLocation: http:\/\/localhost:3005\/public-api\/docs<\/span>\r\n$ curl -sI http:\/\/localhost:3005\/public-api\/docs<\/span>\r\nHTTP\/1.1 200 OK<\/span><\/pre>\n
The application redirects <\/span>\/<\/span> to its doc’s page. It always has. The ALB health check targeted <\/span>\/<\/span> and accepted only <\/span>200<\/span>. That mismatch had existed since the service was deployed.<\/span><\/p>\n
\"\"<\/p>\n
Two health checks, same application, opposite conclusions \u2014 because they probed different paths.<\/span>\u00a0<\/span><\/p>\n
Now the question arises why it fired that day?\u00a0<\/span>\u00a0<\/span><\/h3>\n
The alarm history showed <\/span>INSUFFICIENT_DATA \u2192 ALARM<\/span> with the reason <\/span>“Unchecked: Initial alarm creation.”<\/span><\/i> The alarm had been created that morning as part of a compliance monitoring rollout. The misconfiguration was old; the observation was new.<\/span>\u00a0<\/span><\/p>\n
The fix was to align the ALB health check with the path the liveness probe already used and had already validated:<\/span>\u00a0<\/span><\/p>\n
alb.ingress.kubernetes.io\/healthcheck-path: '\/public-api\/docs'<\/span><\/pre>\n
Targets went healthy in under a minute.<\/span>\u00a0<\/span><\/p>\n
There was a tempting shortcut here: set <\/span>success-codes: “200,302”<\/span> and move on. It would have cleared the alarm just as fast. But it would also mean the ALB considers a target healthy whenever <\/span>anything<\/span><\/i> responds with a redirect \u2014 including a broken application whose redirect middleware still works. The alarm would go quiet while the signal quality got worse.<\/span>\u00a0<\/span><\/p>\n
Clearing an alert and fixing a problem are different objectives.<\/span><\/b> They frequently have different solutions, and the faster one is usually the wrong one.<\/span><\/pre>\n
Let’s have a look at the Comprehensive Diagram:<\/span>\u00a0<\/span><\/h2>\n
\"Flowchart<\/p>\n
Few Important Anecdotes Which I Discussed with Myself Before Starting? <\/span>\u00a0<\/span><\/h2>\n
Map the blast radius before editing anything.<\/span><\/b> A shared <\/span>group.name<\/span> means your one-file change has dependencies in repositories you haven’t opened. Query the live cluster for everything sharing that group before touching the first manifest.<\/span>\u00a0<\/span><\/p>\n
A correct annotation is not a working configuration.<\/span><\/b> IRSA annotations, node labels, and similar settings only take effect at pod creation. A configuration added after the pod started is invisible until something restarts. Verify the <\/span>running process<\/span><\/i>, not just the desired state \u2014 check for the token volume and the environment variables, not the annotation.<\/span>\u00a0<\/span><\/p>\n
Silent fallbacks are the expensive kind of failure.<\/span><\/b> The AWS credential chain is designed never to fail loudly. That’s convenient until it means a workload has been running with the wrong identity for three months. Anywhere a system has a graceful fallback, ask what happens when the primary path silently doesn’t work.<\/span>\u00a0<\/span><\/p>\n
Query the environment before trusting the documentation.<\/span><\/b> Vendor docs can be internally inconsistent. Automated reviewers can confidently state opposite things on consecutive passes. A single <\/span>describe-*<\/span> call is definitive for your account, your region, your resource, right now.<\/span>\u00a0<\/span><\/p>\n
Test persistence for anything applied outside GitOps.<\/span><\/b> If you must configure something imperatively on a controller-managed resource, force a reconcile and confirm it survives. Then document it with a re-apply procedure, because the next person will not know it exists.<\/span>\u00a0<\/span><\/p>\n
Investigate before accepting blame.<\/span><\/b> When an alarm fires shortly after your change, the correlation is suggestive but not conclusive. In this case a five-minute investigation showed a longstanding misconfiguration surfaced by a brand-new alarm. Had I assumed causation, I might have reverted a correct security fix to “resolve” an unrelated problem.<\/span>\u00a0<\/span><\/p>\n
Every genuinely hard bug is at least two bugs.<\/span><\/b> The port-80 change was blocked by an IAM issue, which was compounded by an invalid attribute from the <\/span>other<\/span><\/i> ticket poisoning the same reconciliation loop. Neither would have been especially difficult alone. Interleaved, each one obscured the other’s symptoms.<\/span>\u00a0<\/span><\/p>\n
Conclusion:\u00a0<\/span>\u00a0<\/span><\/h2>\n
The final diff across all of it is about a dozen lines. Two annotations changed, one deleted, one health check path corrected.<\/span>\u00a0<\/span><\/p>\n
What produced those dozen lines was: enumerating an infrastructure boundary that didn’t match the repository boundary, diagnosing an identity fallback that had been silently active for three months, resolving contradictory vendor documentation empirically, and correctly declining to take the blame for an unrelated alarm.<\/span>\u00a0<\/span><\/p>\n
The infrastructure was not broken. It was working exactly as configured. The configuration just encoded several assumptions that had quietly stopped being true\u00a0and nothing in the system was designed to tell anyone.<\/span>\u00a0<\/span><\/p>\n
That’s most of what infrastructure work is. The fix is rarely the hard part.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
This blog aims to share the lessons learned from a deceptively simple security remediation task that turned into a three-day investigation across multiple layers of cloud infrastructure. Through the journey of disabling port 80 on a shared Application Load Balancer (ALB), uncovering a silent IAM role fallback, and identifying an overlooked health check dependency, readers … <\/p>\n
Continue reading “Two-Line Fix Took Three Days: How We Debugged AWS ALB Issues”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":355,"comment_status":"open","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-296","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/posts\/296","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/comments?post=296"}],"version-history":[{"count":18,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/posts\/296\/revisions"}],"predecessor-version":[{"id":361,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/posts\/296\/revisions\/361"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/media\/355"}],"wp:attachment":[{"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/media?parent=296"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/categories?post=296"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.codelogicx.com\/blog\/wp-json\/wp\/v2\/tags?post=296"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}