Pre-upgrade prep

vCenter Server appliance: outbound proxy configuration

Configuring the vCenter Server appliance itself to reach the internet through a proxy (depot access, NTP, DNS-over-internet, external registration, and similar outbound calls) – not to be confused with a VCF Operations Cloud Proxy (a separate appliance for inbound data collection, see Field notes). Applies regardless of track – VCF, VVF, or standalone vSphere – since this is a plain vCenter appliance networking setting, independent of any licensing layer.

The method is version-specific and the two are not interchangeable. Using the pre-9.x file on a 9.x appliance (or vice versa) does not just fail cleanly – on 9.x it is explicitly called out as the wrong file to touch. Check the running vCenter’s major version before following either section below.

Not every appliance component historically read the same setting the same way. Per Broadcom’s own proxy-troubleshooting KB 373713 (applies to vCenter Server 7.0 and later), the VAMI (management interface) uses wget internally while VUM / Lifecycle Manager uses curl – two different HTTP clients that can, in principle, resolve proxy settings from different places (wget also reads /etc/wgetrc / ~/.wgetrc on top of the system-wide setting). Not confirmed by Broadcom as the reason, but worth knowing: on 9.x, proxy handling moves to a single config.json file rather than the older per-tool arrangement, which removes this particular class of inconsistency either way.


vCenter 7.0.x / 8.0.x

Two methods, per Broadcom KB 370265:

VAMI GUI (https://<vcenter-fqdn>:5480 → Networking → Proxy Settings → Edit): pick the traffic type (HTTP/HTTPS/FTP), enter the proxy server/port and optional username/password, Save, then restart services:

service-control --stop --all && service-control --start --all

Manual file edit (SSH to the appliance as root, edit /etc/sysconfig/proxy):

  1. Back it up first: cp proxy proxy.bak
  2. PROXY_ENABLED="yes"
  3. HTTP_PROXY="http://proxy.example.com:8080" and HTTPS_PROXY="http://proxy.example.com:8080" (same URL if it’s a single-port proxy for both)
  4. NO_PROXY="localhost,127.0.0.1,example.com" – CIDR notation is supported here from 7.0 U1c onward
  5. Reboot the appliance (not just a service restart) for the change to take effect

vCenter 9.x

Do not edit /etc/sysconfig/proxy on 9.x – per Broadcom KB 402684, that’s the pre-9.x file and is explicitly the wrong one here.

The VAMI UI’s proxy validation is broken on 9.x – it fails trying to validate the proxy by connecting to vmware.com or the vCenter’s own IP through it, surfacing as “HTTP: Cannot connect to proxy server” even with correct settings. The VAMI UI also flatly rejects CIDR notation in the exclusion list.

Field-observed (2026-09-29, vCenter 9.1): VAMI worked, and was the way out. On this appliance the VAMI UI saved the proxy settings without the validation error above, and it was the only thing that got the proxy service running again after hand-edits had broken it (see Field warning). Try VAMI first. Fall back to the JSON edit below only when VAMI refuses to save, or when you need CIDR in the exclusion list.

How the 9.x proxy works on the appliance

On 9.x, appliance tools don’t talk to the corporate proxy directly. They go through a local proxy service, vmware-envoy-system-proxy, which reads config.json and decides per destination whether to forward to the corporate proxy or connect directly (no_proxy). Field-observed via env | grep -i proxy in the vCenter shell:

VariablePoints to
http_proxyhttp://localhost:1081/
https_proxyhttp://localhost:1082/
ftp_proxyhttp://localhost:1083/
no_proxylocalhost, 127.0.0.1

Three consequences, all seen in the field:

  • Every curl from the shell shows localhost:1082, never the corporate proxy’s name. That’s normal, not a misconfiguration.
  • The shell’s no_proxy is not where exclusions go. It only keeps traffic to the appliance itself away from the local proxy. Exclusions for other hosts belong in config.json (or VAMI).
  • The local proxy can run on stale settings. Plain curl through localhost:1082 got 407 Proxy Authentication Required, while curl -x http://<proxy-host>:8080 – the same proxy, which uses no authentication – returned 200. Only the reboot that followed showed the service couldn’t start with the current file at all. If going through the local proxy and going straight to the corporate proxy give different answers, suspect the local proxy.

Service and file:

systemctl status vmware-envoy-system-proxy --no-pager | head -8
ls -la /var/lib/vmware-envoy-system-proxy/

The service runs as user envoy-system-proxy. On the field appliance config.json was root-owned 644 in a 755 folder, readable by that user. Its request log is under /var/log/vmware/envoy/ (envoy-access*.log, per KB 438438).

The supported path is a direct JSON file edit:

  1. SSH to the vCenter appliance as root.
  2. Edit /var/lib/vmware-envoy-system-proxy/config.json.
  3. One object per protocol – http_proxy, https_proxy, ftp_proxy – each with:
    {
      "scheme": "<proxy url schema>",
      "host": "<proxy url host>",
      "port": <port or null>,
      "username": "<credential or null>",
      "password": "<credential or null>"
    }
    Leave irrelevant protocol entries blank. Remove any comment sections before saving – the file must be valid JSON.
  4. Add a no_proxy array with comma-separated exclusion entries as needed – this is also the only place to add CIDR notation, since the VAMI UI rejects it.
  5. No service restart is needed after saving, per KB 402684. Do one anyway (systemctl restart vmware-envoy-system-proxy), to prove the service accepts the file now rather than at the next reboot – see the Field warning.

Worked example – HTTP and HTTPS traffic through the same proxy, FTP left unused, one internal domain and one CIDR block excluded:

{
  "http_proxy": {
    "scheme": "http",
    "host": "proxy.example.com",
    "port": 8080,
    "username": null,
    "password": null
  },
  "https_proxy": {
    "scheme": "http",
    "host": "proxy.example.com",
    "port": 8080,
    "username": null,
    "password": null
  },
  "ftp_proxy": {
    "scheme": null,
    "host": null,
    "port": null,
    "username": null,
    "password": null
  },
  "no_proxy": ["localhost", "127.0.0.1", "example.com", "10.0.0.0/24"]
}

Validate the file is syntactically correct JSON before trusting it took effect – a trailing comma or leftover comment left in from editing is enough to silently break parsing:

python3 -m json.tool /var/lib/vmware-envoy-system-proxy/config.json

Field warning: a hand-edited config.json can break the proxy service

Field-observed (2026-09-29, vCenter 9.1). Root cause not found. Sequence:

  1. The proxy was disabled temporarily, then re-enabled by editing config.json by hand (a no_proxy exclusion added). Traffic through the local proxy returned 407 from then on (see the stale settings point above).
  2. After a vCenter reboot, vmware-envoy-system-proxy would not start: status=1/FAILURE, nothing useful in journalctl, and every curl failed with Failed to connect to localhost port 1082: Connection refused – no outbound traffic from the appliance at all.
  3. Ruled out: the file was valid JSON (python3 -m json.tool), 644, readable by the service user (su -s /bin/sh envoy-system-proxy -c "cat …"), no credentials set. The service still failed with no_proxy cut to localhost, and with the file renamed away.
  4. Fix: with the file renamed away, the proxy was configured through VAMI, which wrote a new config.json. The service started normally.
  5. A backup of that working file (cp), put back after a further failed hand-edit, failed again. So the contents alone were not the difference. Two suspects remain unconfirmed: systemd’s start limit (too many failed restarts in a row make it refuse, shown as start-limit-hit; cleared with systemctl reset-failed), or something VAMI sets beyond the file itself.

What to do with that:

  • Configure the proxy through VAMI, including the exclusion list, for as long as FQDNs and domains are enough.

  • If you do hand-edit, test right away, not at the next reboot:

    systemctl restart vmware-envoy-system-proxy; sleep 3
    systemctl is-active vmware-envoy-system-proxy
    curl -v -I https://dl.broadcom.com/ 2>&1 | tail -5

    A service that’s still running on old settings hides a broken file until the next boot.

  • Before trusting any repeat test, run systemctl reset-failed vmware-envoy-system-proxy, so a start-limit refusal can’t be mistaken for a bad file.

  • Keep the VAMI-generated file as the reference – rename, don’t delete, when experimenting.

  • A failed proxy service means no internet access for vCenter at all. Lifecycle Manager downloads stop. Anything excluded in no_proxy should be unaffected, if direct access to those hosts is allowed.


Internal VCF components must bypass the proxy

Field-verified (2026-09-29, VVF upgrade to 9.1): with an outbound proxy configured on a vCenter 9.x appliance, assigning a license from VCF Operations failed with:

Failed: The license server SSL certificates are not trusted by the vCenter instance.

The message points at certificates, but the cause was the proxy (the message is a catch-all – it also appears when the certificate push from VCF Operations fails or simply didn’t land; see License Server certificate trust for the flow and a manual trusted-root workaround). vCenter’s traffic to the License Server went through the appliance’s system proxy to the corporate proxy, which couldn’t reach the internal License Server. Correct licensing privileges, and even administrator@vsphere.local as the VCF Operations account, made no difference. Disabling the proxy for the assignment fixed it. No Broadcom KB found that ties this message to the proxy (searched 2026-09-29).

Diagnose it from the vCenter shell – the tell is the tunnel through the local proxy failing (going through localhost:1082 on its own is normal on 9.x, see How the 9.x proxy works):

curl -v https://<license-server-fqdn>/ -o /dev/null
* Host localhost:1082 was resolved.
> CONNECT <license-server-fqdn>:443 HTTP/1.1
* CONNECT tunnel failed, response 502

Then confirm the direct path works by skipping the proxy for one request:

curl -v --noproxy '*' https://<license-server-fqdn>/ -o /dev/null

Connected to … plus a certificate in the output means the License Server is reachable directly and the proxy is the only thing in the way.

Permanent fix: exclude it instead of switching the proxy off. The License Server isn’t contacted only at assignment time; vCenter keeps talking to it, so a proxy re-enabled afterwards can break licensing again. Add the License Server FQDN to the proxy exclusion list – preferably in VAMI; no_proxy in config.json only when you need CIDR, and then read the Field warning first. Ideally exclude the whole internal management domain, which also covers VCF Operations and the VCF Management Services FQDNs.

Not yet verified end-to-end. In the field case, the exclusion was added while the proxy service was in trouble (see the warning above), so “licensing keeps working with the proxy on and the License Server excluded” hasn’t been confirmed on its own.

A plain curl can’t show whether the exclusion works. The shell’s https_proxy always points at the local proxy, and the exclusion is applied inside that service, after curl has handed the request over. So even with the License Server excluded, curl still shows CONNECT <license-server-fqdn>:443 to localhost:1082, followed by:

* CONNECT tunnel established, response 200

(Field-observed 2026-10-13, vCenter 9.1.1.0100, License Server FQDN in the exclusion list.) That 200 is the local proxy saying it opened a tunnel. It doesn’t say whether the next hop was the corporate proxy or the License Server itself. Ways to tell:

  • The certificate in the rest of the same curl -v output. After the CONNECT lines comes the TLS handshake. The Server certificate: issuer should be your VCF or internal CA. A corporate proxy’s CA there means TLS inspection, which is exactly what produces the “not trusted” error above. A 502 instead of 200 means nothing got through at all.

  • curl -v --noproxy '*' (above) for the direct path, to compare with.

  • The local proxy’s access log (/var/log/vmware/envoy/envoy-access*.log): the entry for the License Server FQDN should list the License Server’s own address as upstream, not the corporate proxy’s.

    Untested – the log location is from KB 438438, but which field holds the upstream address hasn’t been checked on an appliance.

    grep -h '<license-server-fqdn>' /var/log/vmware/envoy/envoy-access*.log | tail -3

The final check is in the product: license assignment from VCF Operations succeeds, and the vSphere Client shows no “not connected to a license server” banner.


Lifecycle Manager download sources behind a proxy

vSphere Lifecycle Manager (vLCM) downloads ESX images and patches from four token-based URLs, all through vCenter’s proxy. They replace the old vmware.com / hostupdate.vmware.com sources. Leaving the old ones configured causes the same failures, per KB 390098.

https://dl.broadcom.com/<token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml
https://dl.broadcom.com/<token>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml
https://dl.broadcom.com/<token>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml
https://dl.broadcom.com/<token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml
  • Token: generated in the Broadcom Support Portal. Only users with the Product Administrator role on the site can create one. The token is a secret; keep it out of tickets, screenshots and chat.
  • Where: vSphere Client → Lifecycle Manager → Settings → Administration → Patch Setup. Afterwards the Connectivity Status column should read Connected. Also check that Internet Connectivity is Enabled (vCenter object → Configure → Settings → General).
  • These are host sources only. Updating the vCenter appliance itself through VAMI uses its own URL (KB 390120).
  • The token also drifts between SDDC Manager, Fleet Manager and vCenter; see the field note.

Test the path first, from the vCenter shell

# Through vCenter's own local proxy (what vLCM uses)
curl -v -I https://dl.broadcom.com/ 2>&1 | grep -E "CONNECT|HTTP/|issuer|SSL certificate"

# Straight to the corporate proxy, skipping the local one
curl -v -I -x http://<proxy-host>:<port> https://dl.broadcom.com/ 2>&1 | grep -E "HTTP/|Proxy-Authenticate"

# The real depot index, with the token – expect HTTP 200
curl -v "https://dl.broadcom.com/<token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml" -o /dev/null 2>&1 | grep -E "HTTP/"

If the filtered output is empty, curl failed before getting any response. Re-run without the grep and read the last lines: Connection refused on localhost:1082 means the local proxy service itself isn’t running (see the Field warning).

ResultMeaningFix
Any HTTP/… status from Broadcom, even 403/404 on the bare hostPath works–
CONNECT tunnel failed, response 407Proxy wants authentication. If -x straight to the same proxy works, it’s the local proxy on stale settings, not the corporate proxy (field-observed)Check Proxy-Authenticate:; if the proxy uses none, restart or re-save the proxy config via VAMI
CONNECT tunnel failed, response 403dl.broadcom.com not allowed on the proxyProxy allowlist
CONNECT tunnel failed, response 502Proxy can’t reach the destinationProxy side; for an internal host, exclude it instead
Tries to connect directly, times outNo HTTPS proxy set. vLCM uses the HTTPS proxy for https:// URLs and goes direct if it’s empty (KB 393951)Fill in the HTTPS proxy, even when it’s the same HTTP proxy
wrong version numberHTTPS proxy entered as https:// but the proxy speaks plain HTTP (KB 396787)Scheme http for the HTTPS proxy entry
self-signed certificate in certificate chain, issuer is your own CAProxy does SSL inspection (KB 396511)Exclude dl.broadcom.com from inspection
HTTP 403 from Broadcom on the token URLBad or expired token, or no entitlementNew token; see the field note

The vLCM log shows which proxy it actually used:

grep -h -E "CURLOPT_PROXY|CURLOPT_NOPROXY|curl_easy_perform" /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server*.log | tail -10

Known issue: compatibility data (VCG) can’t be updated behind a proxy

Field-observed (2026-09-29, vCenter 9.1), matching a Broadcom known issue. With the download sources syncing fine, Update compatibility data failed with:

A general system error occurred: “Unable to authenticate with the VMware Verification Service”

KB 438438 describes exactly this: “There is an issue, when using a Proxy for the vCenter, in an online environment. We can’t download the VMware Compatibility Guide - i.e. VCG, but we can update it via sync updates.” Broadcom “is planning to address the issue in a future release.” The compatibility download first requests a login token from auth.esp.vmware.com through the local proxy, and that request fails (503 no_healthy_upstream in /var/log/vmware/envoy/envoy-access*.log), even though curl from the shell reaches the same endpoints.

Confirm it’s this and not a blocked endpoint:

grep -h "auth.esp.vmware.com" /var/log/vmware/envoy/envoy-access*.log | grep -i "no_healthy_upstream" | tail -3
for h in auth.esp.vmware.com vvs.esp.vmware.com vvs.broadcom.com storage.googleapis.com; do
  echo "== $h"; curl -s -o /dev/null -w "%{http_code}\n" https://$h/
done

A 000 for any host means that endpoint still has to be allowed on the proxy/firewall (KB 401192 lists vvs.broadcom.com and storage.googleapis.com; on VCF 9.1 also from the VCF services runtime IP range).

What’s lost until it’s fixed: hardware compatibility checks and reports in vLCM. Image and patch downloads are not affected.

Workaround: load the compatibility data by hand

Broadcom’s workaround is to fetch the compatibility bundle yourself and import it into vLCM’s local database once. After that, Lifecycle Manager → Actions → Sync Updates keeps it current (KB 438438). The procedure is KB 405839 (written for vCenters without internet access); KB 438438 gives the variant for an online vCenter behind a proxy, where everything runs on the vCenter itself.

Untested here. Not yet applied in the field case; the commands are Broadcom’s, from the two KBs. The read -s prompts and the cleanup step are this guide’s additions to keep the client secret out of shell history. Take a snapshot or file-level backup of vCenter first, per KB 401192.

1. Read the vLCM client credentials (vCenter shell, as root):

grep 'client_' /usr/lib/vmware-updatemgr/config/vvs-config.json

Note the client_id and client_secret values. The client secret is a credential: don’t paste it into tickets, chat or scripts you keep.

2a. Download the bundle on the vCenter itself (online variant, KB 438438). The shell’s curl goes through localhost:1082, the path that works, unlike vLCM’s internal route:

read -r  -p "client_id: "     CLIENT_ID
read -rs -p "client_secret: " CLIENT_SECRET; echo
JSON=$(curl -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&grant_type=client_credentials" -X POST https://auth.esp.vmware.com/api/auth/v1/tokens)
TOKEN=$(echo $JSON | jq -r .access_token)
curl -L -H "X-Vmw-Esp-Client: $TOKEN" -X GET "https://vvs.esp.vmware.com/v1/compatible/vcg/bundles/all?format=gz" -o vlcm-vcg-offline.gz

If TOKEN comes back null, the credentials weren’t accepted, or auth.esp.vmware.com isn’t reachable. Check with the endpoint loop above.

2b. Or download it on a Windows admin machine (KB 405839’s PowerShell variant), when the vCenter can’t reach the endpoints at all. The client ID and secret still come from step 1:

$CLIENT_ID     = Read-Host 'client_id'
$CLIENT_SECRET = [Net.NetworkCredential]::new('', (Read-Host 'client_secret' -AsSecureString)).Password   # works on 5.1 and 7
$JSON  = Invoke-WebRequest -Uri "https://auth.esp.vmware.com/api/auth/v1/tokens" -Method POST -Body "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&grant_type=client_credentials"
$TOKEN = ($JSON.Content | ConvertFrom-Json).access_token
Invoke-WebRequest -Uri "https://vvs.esp.vmware.com/v1/compatible/vcg/bundles/all?format=gz" -Method GET -Headers @{"X-Vmw-Esp-Client"="${TOKEN}"} -OutFile vlcm-vcg-offline.gz

Add -Proxy http://<proxy-host>:<port> to both Invoke-WebRequest calls if the workstation needs it explicitly. Then copy the file to the vCenter with scp. The shell must be bash first, see the SCP note at the top of VDT and lsdoctor:

scp .\vlcm-vcg-offline.gz root@<vcenter-fqdn>:/root/

3. Unpack and import (vCenter shell, in the folder holding the file):

gzip -d vlcm-vcg-offline.gz
/usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py update-offline --filePath /<path to the file>/vlcm-vcg-offline

4. Check it took:

/usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py information

Then, in the vSphere Client, run the hardware compatibility check or report that failed before.

5. Clean up and keep it current:

unset CLIENT_ID CLIENT_SECRET TOKEN JSON
rm -f /<path to the file>/vlcm-vcg-offline

From here, Sync Updates refreshes the compatibility data. Only repeat the manual import if the data goes stale again while the known issue is still open. Check the KB for a fixed release before each vCenter update.


Verifying the proxy is actually reachable

Independent of which method configured it, confirm the appliance can reach the internet through the proxy itself before assuming an upgrade-blocking connectivity issue is something else. Per KB 373713, from an SSH session on the appliance:

HTTP_PROXY="http://<proxy-host>:<port>/" curl -I http://example.com
HTTPS_PROXY="https://<proxy-host>:<port>/" curl -I https://example.com
wget --spider http://example.com
https_proxy="http://<proxy-host>:<port>/" wget --spider https://example.com

These test whether the proxy server itself is reachable and forwarding correctly – a clean result here doesn’t by itself confirm vCenter’s own app-level config (config.json on 9.x, /etc/sysconfig/proxy on 7.0.x/8.0.x) is being read correctly, only that the proxy is a working path if the appliance is pointed at it.


Sources

ITQ – What's Next. VCF upgrade-planning material. © 2026 ITQ. hollebollevsan.nl