Product Garaj

WorkAboutLifeContact

PM Portfolio - Project 02

ZTNA Migration at
Enterprise Scale

How I led Fidelity's shift from legacy VPN to zero-trust network access for 15,000+ associates in a regulated financial services environment — cutting mean time to connect by 57% and eliminating $840K in annual VPN infrastructure cost.

Client

Fidelity Investments

Role

Senior PM, Network Infra

Timeline

18 months

Users Migrated

15,200

Domain

ZTNA / SASE / BGP

Period

2022 - 2024

The Routing Problem

Before: VPN Hairpin

User authenticates via VPN client: avg 6m 42s

All traffic forced through DC VPN concentrators before reaching internet

SaaS apps (Bloomberg, Office 365) hairpin through data center: +40-80ms RTT per request

VPN concentrators at 89% capacity during market open

Single BGP path — no per-user policy granularity

Policy change: 3-week change management cycle

After: BGP Microsegmentation + ZTNA

Identity-aware authentication: avg 2m 52s (57% faster)

Direct SaaS path permitted per identity tier — hairpin eliminated

Zero hairpin latency overhead on approved SaaS destinations

DC perimeter traffic reduced 62% — concentrators retired

BGP LOCAL_PREF per community string — per-identity-group routing

Policy change: 4 hours via policy-as-code pipeline

The BGP Policy Abstraction

The core engineering challenge was creating an abstraction between business identity (who you are, what division you work in) and BGP routing policy (which prefixes you can reach, at what LOCAL_PREF). Each identity tier maps to a BGP community string. Route policy at each of 40 DC routers evaluates the community on ingress and applies LOCAL_PREF accordingly. This means a new hire joining the Trading division automatically gets a Tier 2 policy applied to their traffic — no manual firewall rule, no VPN group assignment, no ticket.

Cross-Functional Alignment

Every ZTNA migration dies in committee. The four teams below each had legitimate blocking objections. This is how they were resolved.

InfoSec

Blocking Concern

Owned VPN and its audit trail. ZTNA removes their primary control plane. Needed evidence that BGP route-policy provides equivalent PCI-DSS audit evidence.

Resolution

Built route-policy-as-code pipeline with Git blame trail. Passed PCI-DSS Zone B audit in Phase 3 with zero exceptions.

Network Engineering

Blocking Concern

Concerned about BGP policy explosion — each identity tier adds route-map entries across 40 routers. Worried about routing table instability.

Resolution

Designed community-based model rather than prefix-per-user. 4 community strings cover all 80k associates. BGP table growth: under 2%.

Trading Operations

Blocking Concern

BGP convergence time. Trading floor needs sub-50ms policy propagation. Standard BGP holdtimer is 90 seconds — unacceptable for live trading scenarios.

Resolution

Tuned BGP holdtimer to 3 seconds on trading-adjacent routers. Achieved 8-second worst-case policy propagation. Trading Ops signed off in Phase 4.

Compliance and Legal

Blocking Concern

GDPR data residency constraints for EU trading entities. VPN hairpin enforced data residency by routing everything through London DC. ZTNA breaks this by design.

Resolution

Designed geo-aware BGP community tagging for EU associates. EU-bound traffic still routes through London egress. Non-EU SaaS permitted direct path.

Programme Artefacts

Three views from the live programme: migration phase progress, the ZTNA policy builder showing identity-to-BGP mapping, and the before/after WAN metrics.

ZTNA Programme Dashboard

Fidelity Investments - Enterprise Network Transformation

Phase 4 Active15.2k users

Migration Phases

Policy Builder

WAN Metrics

Total Users

15,200 migrated

Current MTTR

2m 52s

Q2 Target

under 3m 00s

Incidents

0 this quarter

Pilotcomplete8 wks

USERS 500

MTTR 6m 12s

INCIDENTS 2

Established BGP community baseline. InfoSec approved route-policy as audit substitute for VPN logs.

Early Adopterscomplete12 wks

USERS 2,500

MTTR 4m 38s-25%

INCIDENTS 1

Found hairpin on Salesforce via split-tunnel gap. Fixed via policy exception; prevented 22k-user recurrence.

DC East Rolloutcomplete16 wks

USERS 5,000

MTTR 3m 14s-30%

INCIDENTS 0

BGP microsegmentation passed PCI-DSS Zone B audit. First evidence of compliance-by-routing.

4

Trading Deskactive18 wks

USERS 7,000

MTTR 2m 52s-11%

INCIDENTS 0

BGP holdtimer tuned to 3s for sub-50ms policy propagation. Trading Ops signed off. No latency regression.

5

Enterprise Scaleplanned24 wks

USERS 62,500

MTTR target: under 3m

INCIDENTS TBD

Automated BGP policy pipeline in build. Target: policy-change-to-propagation under 4 hours from 3 weeks.

Key PM Decisions

Routing-Layer vs. Firewall-Layer Microsegmentation

InfoSec initially wanted microsegmentation enforced at the firewall layer (Palo Alto rules per identity group). Network Engineering proposed BGP-level enforcement. PM had to arbitrate.

Decision Made

BGP routing layer enforcement via LOCAL_PREF and community strings.

Tradeoff Accepted

Accepted higher BGP operational complexity. Mitigated by automated policy pipeline and route-policy as code (stored in Git, reviewed by InfoSec before deploy).

Outcome

Policy change time: 3 weeks to 4 hours. Zero compliance exceptions on PCI-DSS audit.

Phased Rollout vs. Big-Bang Migration

Project sponsor wanted an 8-week full enterprise cutover to hit a board-level security milestone. PM assessed this as 8x the risk for a 2x timeline gain.

Decision Made

5-phase rollout over 18 months, division by division.

Tradeoff Accepted

Delayed full ZTNA benefit by 12 months. Required sustained stakeholder management over a longer horizon. Project sponsor required monthly SLA briefings to maintain confidence.

Outcome

Zero production incidents affecting end users. BGP policy models validated by live load before scaling.

Measured Outcomes

57%

reduction in mean time to connect — from 6m 42s to 2m 52s across 15,200 migrated associates

Fidelity network telemetry, Q3 2023 vs Q1 2024

authentication-related incidents in two quarters post-migration, down from 12 per quarter with VPN

Fidelity NOC incident log

$840K

annual OPEX eliminated by retiring VPN concentrators and associated licensing across 4 data centers

Network infrastructure cost model, 2024

4 hrs

to deploy new BGP route policy across all data centers — reduced from 3 weeks via policy-as-code pipeline

Change management records, 2024

Why This Matters for Cloudflare WAN

The Same Abstraction Problem, at Internet Scale

The policy abstraction I solved at Fidelity — mapping identity-aware business rules to BGP routing config without requiring network expertise — is exactly what Cloudflare One builds at global scale. The operator problem is identical: enterprises need to express zero-trust policy in business terms, not in routing syntax. At Cloudflare, policy propagates across an Anycast network in under 50ms worldwide rather than across 40 DC routers in under 60 seconds. The PM challenge is the same; the scale is three orders of magnitude larger.

VPN Hairpin is the Market Problem Cloudflare Sells Against

Every Cloudflare One sales conversation starts with the same whiteboard: the legacy VPN hairpin diagram. I have lived this problem from the PM side — the $840K in concentrator cost, the 6-minute authentication latency, the PCI-DSS compliance friction, the BGP holdtimer tuning on trading desks. I can walk into a Cloudflare WAN customer conversation and speak to every objection from InfoSec, Network Engineering, Compliance, and Trading Ops because I have resolved all of them in production.

Product Portfolio - Rebecka Raj

18 months, 15,200 users, 40 data center routers. A regulated-industry ZTNA migration from first pilot to trading desk.

Back to Projects