Azure Application Gateway: Fix Health Probe Failures Due to Backend Port Mismatch
Quick Fix Summary
TL;DRVerify and correct the backend port in the HTTP settings and health probe configuration.
Health probes fail when the port configured in the Application Gateway's HTTP settings or health probe does not match the actual port where the backend service is listening.
Diagnosis & Causes
Recovery Steps
Step 1: Verify Current Configuration and Probe Status
Check the current backend port and health probe settings, and view the health status of backend pools.
# Get Application Gateway details
az network application-gateway show --name <gateway-name> --resource-group <resource-group-name> --query '{BackendPorts:backendHttpSettingsCollection[].port, ProbePorts:probes[].port, BackendHealth:backendAddressPools[].backendHttpSettings.backendServers}' -o json
# Check backend health (requires gateway to be running)
az network application-gateway show-backend-health --name <gateway-name> --resource-group <resource-group-name> -o tableStep 2: Correct Backend Port in HTTP Settings
Update the backend port for the relevant HTTP settings object to match your backend service port (e.g., 8080).
az network application-gateway http-settings update --gateway-name <gateway-name> --resource-group <resource-group-name> --name <http-settings-name> --port <correct-backend-port>Step 3: Correct Health Probe Port (If Different from HTTP Settings)
If your health probe uses a specific port (not 'null' which inherits from HTTP settings), update it to match the backend service.
az network application-gateway probe update --gateway-name <gateway-name> --resource-group <resource-group-name> --name <probe-name> --port <correct-backend-port>Step 4: Verify Backend Service is Listening
Connect to a backend VM/instance and confirm the service is running and listening on the expected port.
# On a Linux backend (example for port 8080)
sudo netstat -tlnp | grep :8080
# Or using ss
sudo ss -tlnp | grep :8080
# Check service status (adjust for your service)
sudo systemctl status <your-backend-service>Step 5: Test Network Connectivity from Gateway Subnet
Simulate a health probe from within the Application Gateway's subnet to the backend IP:Port using a test VM.
# On a test VM in the Gateway's subnet (V2 SKU: GatewayManager service tag)
# Install netcat/telnet if needed
sudo apt-get install netcat
# Test TCP connectivity
nc -zv <backend-private-ip> <backend-port>
# Test HTTP probe (if protocol is HTTP/HTTPS)
curl -I http://<backend-private-ip>:<backend-port><probe-path>Step 6: Review NSG and Firewall Rules
Ensure Network Security Groups (NSGs) and any host firewalls allow traffic from the Application Gateway's subnet to the backend port.
# Check NSG rules on the backend subnet or NIC
az network nsg rule list --nsg-name <nsg-name> --resource-group <resource-group-name> -o table
# For V2 SKU, source is 'GatewayManager' service tag. For V1, it's the Gateway subnet address range.Architect's Pro Tip
"This often happens after migrating workloads or changing default ports. The health probe port defaults to 'null', inheriting from HTTP settings. Explicitly setting a probe port overrides this, creating a mismatch if not updated alongside HTTP settings."
Frequently Asked Questions
How do I find the correct HTTP settings or probe name to update?
Run 'az network application-gateway show' and look at the 'backendHttpSettingsCollection' and 'probes' arrays. The 'name' field is used in the update commands. The probe is linked to the HTTP settings.
My backend uses a custom port like 3000 or 8080. What should the probe path be?
The probe path is defined relative to the backend application root on that specific port. Ensure the path (e.g., '/' or '/health') returns a 200 OK status code on the custom port, not just the default port 80.