<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Peakhour.IO - Incident Response</title><link href="https://www.peakhour.io/" rel="alternate"></link><link href="https://www.peakhour.io/feeds/tag/incident-response.atom.xml" rel="self"></link><id>https://www.peakhour.io/</id><updated>2026-08-01T11:50:00+10:00</updated><entry><title>What a Residential Proxy Decision Log Should Contain</title><link href="https://www.peakhour.io/blog/residential-proxy-decision-logging/" rel="alternate"></link><published>2026-08-01T11:50:00+10:00</published><updated>2026-08-01T11:50:00+10:00</updated><author><name>AC</name></author><id>tag:www.peakhour.io,2026-08-01:/blog/residential-proxy-decision-logging/</id><summary type="html">&lt;p&gt;A useful proxy event records the request, signal provenance, policy, action, and outcome. A timestamp and a risk score are not enough to explain an enforcement decision.&lt;/p&gt;</summary><content type="html">&lt;p&gt;A log line that says &lt;code&gt;proxy=true&lt;/code&gt; is intelligence. It is not a decision record.&lt;/p&gt;
&lt;p&gt;When a customer asks why a login was challenged, a fraud analyst reviews an account event, or an engineer tunes a rate limit, the team needs to reconstruct what the control knew and what it did. A timestamp, IP address, and final risk score cannot answer that on their own.&lt;/p&gt;
&lt;p&gt;&lt;a href="/products/residential-proxy-detection/"&gt;Peakhour Residential Proxy Detection&lt;/a&gt; keeps the proxy result inside the request decision with IP, credential, device, behaviour, and route context. The record should preserve that connection without becoming an indiscriminate collection of personal data.&lt;/p&gt;
&lt;h2&gt;Record the Request Being Judged&lt;/h2&gt;
&lt;p&gt;The event comes first. The detector explains one part of it.&lt;/p&gt;
&lt;p&gt;Useful request fields include:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;observed_at
request_id and trace_id
service, host and route class
HTTP method
response status
account, tenant, session or API-key reference where permitted
edge location and origin target
cache outcome and whether origin was reached
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Avoid logging secrets, complete session tokens, passwords, payment data, or unrestricted request bodies. The &lt;a href="https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html"&gt;OWASP Logging Cheat Sheet&lt;/a&gt; recommends excluding or masking access tokens, authentication passwords, sensitive personal data, and other values whose collection creates more risk than operational value.&lt;/p&gt;
&lt;p&gt;Use stable references where the investigation needs continuity, and keep their purpose and retention bounded.&lt;/p&gt;
&lt;h2&gt;Keep Signal Provenance&lt;/h2&gt;
&lt;p&gt;The value of a signal depends on how it was produced.&lt;/p&gt;
&lt;p&gt;For proxy and network evidence, retain:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;proxy_result and category
detector or feed name
detector version or dataset time
network type, ASN and country
client IP source and trusted forwarding hop
fingerprint method, version and capture point
browser or device evidence source
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A fingerprint without a method and capture point cannot be compared safely. A country value without a source and observation time is difficult to audit after data changes. A forwarded IP taken from an untrusted header should never have reached policy in the first place.&lt;/p&gt;
&lt;p&gt;Flattening every input into one number leaves the analyst with a threshold and no explanation. Retain the factors that moved the score.&lt;/p&gt;
&lt;h2&gt;Record the Policy and the Action&lt;/h2&gt;
&lt;p&gt;The event should identify the control that made the decision:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;policy_id and revision
policy owner
mode: monitor or enforce
matched conditions
threshold and current counter state
selected action
action reason code
expiry or review date
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Reason codes should be stable enough to aggregate. Free-text analyst notes can add context, but they should not be the only explanation.&lt;/p&gt;
&lt;p&gt;The action field needs more detail than &lt;code&gt;blocked=true&lt;/code&gt;. Distinguish allow, log, challenge, rate limit, deny, route, and step-up outcomes. For a limit, keep the key class and interval without exposing sensitive raw values. For a challenge, record whether it was presented, completed, failed, expired, or abandoned.&lt;/p&gt;
&lt;h2&gt;Connect the Decision to Its Outcome&lt;/h2&gt;
&lt;p&gt;Security controls are judged by what happened next.&lt;/p&gt;
&lt;p&gt;Join the decision to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;successful login, signup, reset, checkout, or API completion;&lt;/li&gt;
&lt;li&gt;authentication failure and account lockout;&lt;/li&gt;
&lt;li&gt;challenge completion and abandonment;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;429&lt;/code&gt;, deny, or step-up response;&lt;/li&gt;
&lt;li&gt;origin request avoided or origin work created;&lt;/li&gt;
&lt;li&gt;support case, analyst disposition, or confirmed incident where available;&lt;/li&gt;
&lt;li&gt;policy rollback or exception.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is how a team distinguishes attack reduction from customer suppression. If challenges rise and successful account recovery falls, the proxy policy may be adding friction in the wrong place.&lt;/p&gt;
&lt;p&gt;NIST SP 800-92, the &lt;a href="https://csrc.nist.gov/pubs/sp/800/92/final"&gt;Guide to Computer Security Log Management&lt;/a&gt;, treats log management as a lifecycle of generating, transmitting, storing, analysing, and disposing of data. That lifecycle matters here. A detailed event that nobody can query, retain safely, or delete on schedule is not operational evidence.&lt;/p&gt;
&lt;h2&gt;Metrics Should Lead to a Decision&lt;/h2&gt;
&lt;p&gt;For an enforcement platform, useful measures should connect signals to route outcomes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;proxy prevalence by route and action;&lt;/li&gt;
&lt;li&gt;percentage of proxy-labelled traffic allowed, challenged, limited, and blocked;&lt;/li&gt;
&lt;li&gt;challenge completion for proxy and non-proxy cohorts;&lt;/li&gt;
&lt;li&gt;legitimate transaction completion after action;&lt;/li&gt;
&lt;li&gt;confirmed abuse caught with and without a proxy signal;&lt;/li&gt;
&lt;li&gt;false-positive and analyst-overturn rates;&lt;/li&gt;
&lt;li&gt;origin requests and application cost avoided;&lt;/li&gt;
&lt;li&gt;time from observation to policy change;&lt;/li&gt;
&lt;li&gt;rules past their review date.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A count of “risky sessions detected” is hard to act on. A rise in residential-proxy password-reset attempts with low challenge completion and repeated account failures can justify a route-specific change.&lt;/p&gt;
&lt;h2&gt;A Minimal Event&lt;/h2&gt;
&lt;p&gt;A compact decision record might resemble:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;request_id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;req_...&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;route_class&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;account_recovery&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;proxy&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;result&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;category&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;residential&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;source&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;resip&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;fingerprint&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;method&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ja4&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;version&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pinned&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;capture&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;client_edge&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;credential_state&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;exposed_match&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;recent_outcome&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;repeated_failure&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;policy&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;recovery-risk&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;revision&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;mode&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;enforce&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;action&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;step_up&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;reason_codes&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;residential_proxy&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;exposed_credential&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;failure_velocity&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;outcome&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pending&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The exact schema will differ by application. The contract should not: current request, signal provenance, policy revision, selected action, and measurable outcome.&lt;/p&gt;
&lt;p&gt;The schema is ready when a support engineer can explain a challenge, an analyst can reconstruct an account event, and the policy owner can measure the result without opening a second system. Anything collected beyond that needs a stated security purpose and retention period.&lt;/p&gt;</content><category term="Security"></category><category term="Residential Proxies"></category><category term="Security Logging"></category><category term="SIEM"></category><category term="Bot Management"></category><category term="Incident Response"></category></entry><entry><title>Cloudflare outage proves Plan B depends on controlling DNS</title><link href="https://www.peakhour.io/blog/cloudflare-outage-dns-plan-b/" rel="alternate"></link><published>2025-11-18T21:00:00+11:00</published><updated>2025-11-18T21:00:00+11:00</updated><author><name>Dan</name></author><id>tag:www.peakhour.io,2025-11-18:/blog/cloudflare-outage-dns-plan-b/</id><summary type="html">&lt;p&gt;Tuesday’s Cloudflare incident reminded everyone that you can’t execute a Plan B if your DNS knobs are trapped behind the provider that’s failing. Here’s how Peakhour runs a detect-decide-divert playbook without touching your existing third-party DNS vendors.&lt;/p&gt;</summary><content type="html">&lt;p&gt;On Tuesday, 18 November 2025, Cloudflare’s own status page marked every major service—CDN, Firewall, WARP, Workers, and the dashboard—as degraded for most of the day while engineers worked through an internal control-plane failure. The timeline moved from “Investigating” at 11:48 UTC to “Monitoring” after 14:42 UTC, and the incident wasn’t officially resolved until 19:28 UTC. During the worst of it, Cloudflare disabled WARP in London, bot scores seesawed, and customers were told to wait while remediation continued.&lt;/p&gt;
&lt;p&gt;Waiting was the only option for many teams because their Plan B lived behind the same dashboard that was timing out. The top comment on the Hacker News thread was a set of &lt;code&gt;curl&lt;/code&gt; commands for moving domains off Cloudflare’s proxy edge. Admins were stuck in 2FA flows trying to fetch an API token, or searching for Terraform credentials so they could toggle a proxied flag. That is not a resilience strategy.&lt;/p&gt;
&lt;p&gt;We learned this lesson the hard way—and wrote about it after the 2021 Fastly outage in &lt;a href="/blog/fastly-outage-how-to-have-a-plan-b"&gt;How to have a Plan B&lt;/a&gt;. The rule still stands: the platform you are trying to leave cannot be the only place that can change where your DNS points.&lt;/p&gt;
&lt;h2&gt;Detect: understand what’s actually broken&lt;/h2&gt;
&lt;p&gt;Incidents like Tuesday’s change shape quickly. Cloudflare’s own feed showed different failure domains every 30 minutes: bot management, dashboard auth, Access, WARP. The first mile is impartial telemetry that tells you what your users feel, not what the provider thinks. At Peakhour we stream real user monitoring, synthetic checks, and control-plane health from multiple CDNs and DNS partners. That lets us distinguish “cache errors in Hong Kong” from “global auth outage” and choose the right lever.&lt;/p&gt;
&lt;h2&gt;Decide: keep DNS authority in neutral territory&lt;/h2&gt;
&lt;p&gt;When your domain delegation lives with agnostic providers—Route 53, NS1, Azure DNS, or the enterprise registrar your legal team already approved—you can make failover decisions without pleading with a failing control plane. Peakhour doesn’t replace those vendors; we orchestrate them. We set short-but-safe TTLs, keep secondary answers staged, and continuously audit API access so we can flip traffic with one signed request. The minute you outsource DNS authority to a proxy CDN, you have given up the control that makes Plan B possible.&lt;/p&gt;
&lt;h2&gt;Divert: run the playbook in minutes, not hours&lt;/h2&gt;
&lt;p&gt;A workable Plan B has three moves:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Pre-stage alternate edges.&lt;/strong&gt; Your secondary CDN, origin, or transit provider must be in sync with the active one—certificates, cache rules, WAF policies, everything. We keep them hot by replaying production configs across vendors.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Wire DNS automation.&lt;/strong&gt; We integrate with multiple third-party DNS APIs at once so we can update apex A/AAAA, flattened CNAMEs, and geo/latency rules in a single workflow. Because the automation lives off the impacted platform, we can execute even while Cloudflare’s dashboard is returning 500s.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Drill humans on the handoff.&lt;/strong&gt; Our SOC sits in Sydney and Melbourne, but we cover global hours. During an incident we line up Slack/Teams bridges with your SREs, confirm business impact, and keep execs in the loop while traffic drains to the healthy provider.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;With that in place we routinely hit sub-five-minute diversion times, including DNS propagation, because the decision, the tooling, and the people are ready before the outage hits.&lt;/p&gt;
&lt;h2&gt;What Peakhour brings to your Plan B&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Independent authority, familiar vendors.&lt;/strong&gt; We leverage multiple established DNS providers instead of locking you into ours. You keep your contracts; we bring the automation and guardrails.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Unified multi-CDN config.&lt;/strong&gt; Cache rules, image optimisation, WAF, and routing policies stay aligned across providers so you don’t lose capabilities when you switch.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Real drills, not just runbooks.&lt;/strong&gt; Quarterly failover exercises prove that certificates, APIs, and humans are ready. We share the post-mortems so your execs see clear RTO/RPO numbers.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;People you can phone.&lt;/strong&gt; 24×7 Australian-based engineers who know your stack and can execute the play while your own team communicates with customers.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Book a resilience review&lt;/h2&gt;
&lt;p&gt;If Tuesday exposed that your failover path still depends on your primary provider’s dashboard, book a 30-minute Resilience Review with Peakhour and we’ll:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Map who really controls your DNS today.&lt;/li&gt;
&lt;li&gt;Identify the gaps between your primary and standby CDNs.&lt;/li&gt;
&lt;li&gt;Outline the automations we can layer on top of your existing DNS and hosting vendors.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The output is a concrete Plan B, a drill schedule, and a team that can execute it the next time a global provider blinks.&lt;/p&gt;</content><category term="Interest"></category><category term="CDN"></category><category term="DNS"></category><category term="Multi CDN"></category><category term="Incident Response"></category></entry></feed>