Troubleshooting Hybrid Cloud Access: CDN 403 Forbidden Errors When Serving Content from On-Premises OSS Origin
Quick Fix Summary
TL;DRVerify the CDN domain's origin configuration and ensure the on-premises OSS gateway is reachable and authorized.
A 403 error occurs when the CDN edge node is denied access to the origin server. In a hybrid setup, this is typically due to misconfigured access control on the on-premises OSS gateway, incorrect CDN origin settings, or network path issues.
Diagnosis & Causes
Recovery Steps
Step 1: Verify CDN Logs and Isolate the Layer
Check the CDN real-time logs for the specific 403 request. Identify if the error is from the CDN cache (OSS_403) or a direct origin fetch failure. This isolates the problem to CDN configuration vs. origin server.
# Use Alibaba Cloud CLI to pull recent logs for your domain and error code.
aliyun cdn DescribeCdnDomainLogs --DomainName your-cdn-domain.com --StartTime '2024-01-01T00:00:00Z' --EndTime '2024-01-01T01:00:00Z' --PageSize 50
# Filter logs for 403 status in the 'OriginStatus' field (origin error) or 'Status' field (cache error).
grep -E "(\"Status\":403|\"OriginStatus\":403)" cdn-logs.jsonStep 2: Validate CDN Origin Configuration
Ensure the CDN origin settings point to the correct on-premises OSS gateway endpoint and use the proper authentication method. Verify the AccessKey or STS token is valid and has `oss:GetObject` permissions.
# Describe the CDN domain's configuration.
aliyun cdn DescribeCdnDomainDetail --DomainName your-cdn-domain.com
# Pay close attention to the 'Sources' block, looking for 'Origin' (your gateway address), 'Type' (oss), and 'Priority'.Step 3: Check On-Premises OSS Gateway Access Control
The most common cause. The on-premises OSS gateway must whitelist the public IP ranges of Alibaba Cloud CDN edge nodes. Verify and update the gateway's IP ACL or security group rules.
# 1. Identify the requesting CDN edge IP from Step 1 logs (client IP).
# 2. On your on-premises gateway/firewall, verify the rule. Example for iptables:
sudo iptables -L -n | grep <CDN_EDGE_IP>
# 3. Add the official Alibaba Cloud CDN back-to-origin IP ranges to your allow list. Get ranges from:
# https://help.aliyun.com/document_detail/27217.htmlStep 4: Test Direct Origin Access from a CDN Region
Simulate a CDN origin pull by using `curl` from an Alibaba Cloud ECS instance (in a relevant region) to your on-premises OSS gateway URL. This bypasses CDN to test the origin path directly.
# On an Alibaba Cloud ECS instance, test access to the origin object.
# Use the exact origin URL configured in CDN, including any signature parameters.
curl -v -H "Host: your-onprem-oss-host.com" \
"http(s)://your-onprem-gateway-ip/bucket/object?OSSAccessKeyId=...&Signature=..."Step 5: Review and Rotate Authentication Credentials
If using AccessKey or STS in the origin URL, the signature may have expired or be incorrect. For production, use RAM roles (STS) for automatic credential rotation. Manually check key validity and permissions.
# Check if the RAM user/role associated with the AccessKey has the correct policy.
aliyun ram ListPoliciesForUser --UserName <RAM_User_Name>
# For a more secure setup, attach a policy like:
# { "Version": "1", "Statement": [{ "Effect": "Allow", "Action": "oss:GetObject", "Resource": "acs:oss:*:*:*/*" }] }Step 6: Inspect On-Premises Gateway Logs
Examine the access logs on your on-premises OSS-compatible gateway (e.g., MinIO, Ceph RGW). Look for the denied request from the CDN IP and the specific reason code for the 403 (e.g., SignatureDoesNotMatch, AccessDenied).
# Example for MinIO server logs (typically JSON). Find the entry for the CDN IP.
sudo tail -f /var/log/minio/log.json | grep <CDN_EDGE_IP> | grep -i "403"
# The log entry will contain an 'error' field detailing the cause.Architect's Pro Tip
"This often happens after a CDN service upgrade or expansion where new edge node IPs are added to the pool. Your on-premises whitelist becomes stale. Subscribe to Alibaba Cloud's IP range change notifications and automate your ACL updates."
Frequently Asked Questions
Why do I see 403 only sometimes, not for all requests?
Intermittent 403s usually point to an IP whitelist issue. CDN uses a pool of edge nodes. A request routed through a new, non-whitelisted node IP will fail, while others succeed. It can also indicate an expiring STS token that is periodically renewed.
Should I use 'OSS Domain' or 'Custom Origin' type in CDN for my on-premises gateway?
Use 'Custom Origin'. The 'OSS Domain' type is optimized for Alibaba Cloud's native OSS and may apply internal authentication. 'Custom Origin' gives you full control over the origin server address and allows you to pass the necessary authentication headers/parameters to your on-premises gateway.