Troubleshooting & health checks

Start here

Run the bundled script. It works through the same ladder a human would, in the order that isolates faults fastest, and stops at the first thing that is actually broken.

export WRF_TEST_CLIENT_ID='…'
export WRF_TEST_CLIENT_SECRET='…'

./health-check.sh test

Only bash and curl are required. Credentials are read from the environment so they never enter shell history. Add --write to also exercise a real create and update; on production that additionally needs --force-prod, because the probe creates a node that the API cannot delete afterwards.

Is it the WAF or is it Drupal?

This matters more than any token question, because a WAF 403 and a Drupal 403 are the same status code with unrelated fixes — and the WAF one is not yours to fix.

SuiteScript's N/https cannot execute JavaScript, so it can never solve a challenge. If NetSuite's calls are being challenged, the integration is blocked at the edge and credential rotation will not help. The paths that need a WAF skip rule are:

/api*                       (the on-site Swagger UI page)
/openapi/*                  (the spec that page loads)
/oauth/*
/.well-known/jwks.json      (/oauth/jwks redirects here)
/node
/node/*
/taxonomy/term
/taxonomy/term/*
/entity/node_type/*

Full remediation runbook → Who owns the fix, which Cloudflare control to change, the exact rule expression, and an interim path that works today.

Check it in one command:

curl -sSL -D - -o /dev/null 'https://e3test.waterrf.org/oauth/jwks' | grep -iE 'cf-mitigated|^HTTP'

cf-mitigated: challenge present → edge problem, stop debugging tokens. Absent, with HTTP/2 200 → the edge is clear and everything below applies.

Unblocking a Cloudflare 403 — runbook

Ownership matters here: Cloudflare is the authoritative DNS provider and reverse proxy for waterrf.org, so this is WRF's own Cloudflare zone. Acquia Support cannot fix it — the change is made by whoever administers that zone.

Step 1 — Capture the evidence engineer · 2 min

The zone admin cannot find the event without the ray ID. Collect all four:

# Ray ID, status, and the cf-mitigated header
curl -sS -D - -o /dev/null 'https://e3test.waterrf.org/oauth/jwks' \
  | grep -iE '^HTTP|^cf-ray|^cf-mitigated|^server'

# The egress IP Cloudflare actually saw
curl -sS https://api.ipify.org; echo

# UTC timestamp, to match the Security Events log
date -u '+%Y-%m-%dT%H:%M:%SZ'

From NetSuite, the egress IP is not your workstation's — it is the NetSuite data centre's. Get it from the failing SuiteScript's log or from your NetSuite administrator.

Step 2 — Identify which Cloudflare feature fired zone admin · 5 min

Do not guess at the fix. In the Cloudflare dashboard for the waterrf.org zone: Security → Events, filter by Ray ID using the cf-ray value from step 1. The event row names the service that acted, and that determines which control to change:

Step 3 — Choose the allow mechanism decision

Step 4 — Create the rule zone admin · 10 min

Security → WAF → Custom rules → Create rule. Name it something a future admin will understand, e.g. Skip bot checks for NetSuite API. Expression:

(http.host in {"www.waterrf.org" "e3test.waterrf.org" "sandbox.waterrf.org"})
and (
  starts_with(http.request.uri.path, "/api")
  or starts_with(http.request.uri.path, "/openapi/")
  or starts_with(http.request.uri.path, "/oauth/")
  or http.request.uri.path eq "/.well-known/jwks.json"
  or http.request.uri.path eq "/node"
  or starts_with(http.request.uri.path, "/node/")
  or starts_with(http.request.uri.path, "/taxonomy/term")
  or starts_with(http.request.uri.path, "/entity/node_type/")
)

Ready-to-apply versions of this rule — dashboard, Cloudflare API, and Terraform — are in cloudflare/ in this repo, together with the verified table of which paths sit under /api* and which do not.

Action Skip, then tick the components the step 2 event named — typically Super Bot Fight Mode, Managed Rules, and Browser Integrity Check. Place the rule first in the custom-rules order, or a rule above it will still challenge.

Using an IP list: Manage Account → Configurations → Lists, create netsuite_egress, then reference it as $netsuite_egress. NetSuite's outbound ranges vary by account data centre and change over time — get the authoritative list from your NetSuite administrator, not from a blog post, and set a reminder to re-verify it.

Step 5 — Verify engineer · 1 min

./health-check.sh test

Check 1 should flip from FAIL … Blocked by the Cloudflare WAF to:

1. Reachability — GET /oauth/jwks (no credentials)
  PASS  Site up, routing intact, OAuth key material deployed.

Then confirm the header is gone — no cf-mitigated should appear:

curl -sS -D - -o /dev/null 'https://e3test.waterrf.org/oauth/jwks' | grep -i 'cf-mitigated' || echo "clear"

Cloudflare rule changes propagate in seconds, but a cached challenge cookie can mask success in a browser — retest in a private window.

Step 6 — Interim: work via the Acquia origin engineer · now

Cloudflare is not in front of the Acquia origin hostnames, so they answer normally while the public hostnames are being challenged. Verified 2026-09-01 against the staging origin: /.well-known/jwks.json returned 200 with real keys, and /oauth/token returned a proper OAuth error for bad credentials — so the whole auth path is testable.

./health-check.sh origin        # staging, Cloudflare bypassed

# or directly
curl -sS -X POST 'https://wrfsite1stg.prod.acquia-sites.com/oauth/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'grant_type=client_credentials' \
  -d "client_id=$WRF_TEST_CLIENT_ID" \
  -d "client_secret=$WRF_TEST_CLIENT_SECRET" \
  -d 'scope=default'

Step 7 — If the rule cannot be changed

If the zone admin will not skip bot checks on those paths, option C is the fallback: NetSuite sends a shared secret header and the rule matches on it instead of on an IP.

any(http.request.headers["x-wrf-integration"][*] eq "<long-random-value>")

Weaker than mTLS — the value appears in Cloudflare logs — so rotate it on a schedule and keep it out of tickets. Cloudflare Access service tokens (CF-Access-Client-Id / CF-Access-Client-Secret) are the hardened version of the same idea.

Status code reference

Token problems

“It worked five minutes ago”

Access tokens are configured to expire after 300 seconds. This is the single most common cause of an integration that intermittently returns 401. A scheduled script that fetches one token and then works through a long batch will cross the boundary mid-run.

“Refresh token doesn't work”

It cannot. The default scope has refresh_token disabled and only client_credentials enabled, so no refresh token is ever issued. Re-authenticate instead — it is a single cheap call.

“Credentials are right but I get invalid_client”

Check which environment you are pointed at. Test uses NetSuite sandbox 9336228-SB1, production uses 9336228, and the Drupal consumers are separate records with separate secrets. A prod secret against test fails exactly like a revoked secret.

“Writes return 403 but reads work”

The token is fine; content permissions are not. The default scope is role-bound to administrator, so the consumer's user must hold that role. Confirm with GET /oauth/debug?_format=json — if roles does not list administrator, that is the bug.

“Every sync creates duplicates instead of updating”

Updates address nodes by Drupal nid, not by field_netsuite_id. Persist the nid returned by POST /node on the NetSuite record and use it for later PATCH calls. There is no upsert-by-external-id behaviour.

Health check ladder

Each rung narrows the fault. Run them in order; the first failure is the one worth investigating. health-check.sh automates all of this.

1. 1 — Is the edge letting me through? (no credentials)

   curl -sSL -D - -o /dev/null 'https://e3test.waterrf.org/oauth/jwks' | grep -iE '^HTTP|cf-mitigated'
   # want: HTTP/2 301 then HTTP/2 200   (this path redirects to /.well-known/jwks.json)
   # bad:  HTTP/2 403 with cf-mitigated: challenge
   # Use -L. Without it a healthy site looks like a bare 301 "failure".

2. 2 — Can I get a token?

   TOKEN=$(curl -sS -X POST 'https://e3test.waterrf.org/oauth/token' \
     -H 'Content-Type: application/x-www-form-urlencoded' \
     -d "grant_type=client_credentials" \
     -d "client_id=$WRF_TEST_CLIENT_ID" \
     -d "client_secret=$WRF_TEST_CLIENT_SECRET" \
     -d 'scope=default' | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
   
   echo "${TOKEN:0:24}…"

3. 3 — Does Drupal accept the token? (the authentication proof)

   curl -sS 'https://e3test.waterrf.org/oauth/userinfo' -H "Authorization: Bearer $TOKEN"

4. 4 — Does the token carry the right role?

   curl -sS 'https://e3test.waterrf.org/oauth/debug?_format=json' -H "Authorization: Bearer $TOKEN"
   # want: "roles" includes "administrator"
   # 403 here only means the debug permission is missing — harmless for the integration

5. 5 — Is REST serving JSON? (not an auth check)

   curl -sS -o /dev/null -w '%{http_code}\n' \
     'https://e3test.waterrf.org/entity/node_type/resource?_format=json' \
     -H "Authorization: Bearer $TOKEN"
   # 200 = the request reached Drupal and REST works.
   # It does NOT prove your token is valid: this resource is readable
   # ANONYMOUSLY (verified 2026-09-01). Step 3 is the authentication proof.
   # But: userinfo 200 + write 403 does isolate the fault to content permissions.

6. 6 — Control: prove _format is mandatory

   curl -sS -o /dev/null -w '%{http_code}\n' \
     'https://e3test.waterrf.org/entity/node_type/resource' \
     -H "Authorization: Bearer $TOKEN"
   # 406 is the correct answer here. Getting 406 elsewhere means you dropped the parameter.

7. 7 — Can it write? (leaves an unpublished node behind)

   curl -sS -X POST 'https://e3test.waterrf.org/node?_format=json' \
     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
     -d '{"type":[{"target_id":"resource"}],
          "title":[{"value":"ZZZ API health check"}],
          "status":[{"value":false}]}'

   DELETE is not enabled on the node resource, so the node has to be removed by hand at /node/<nid>/delete. Do this on test.

Where to look on the Drupal side

What to send when escalating

Include all of this — without it the first reply will just ask for it:

• Environment (test or prod) and the exact URL, including the query string.
• HTTP method and status code.
• Full response headers. This is what separates a WAF 403 from a Drupal 403.
• Response body, verbatim. The message field usually contains the answer.
• Request body with secrets redacted — the field shapes matter.
• UTC timestamp, so it can be correlated against the Drupal log and Cloudflare events.
• The cf-ray header if present — Cloudflare support needs it.
• Output of ./health-check.sh <env>.