Network Design: Circuit, Active/Active BGP, and other Network Edge Design Considerations
- Sep 13
- 25 min read
Updated: Nov 9
Follow up to the original Dual ISP, DMZ, and the Network Edge post, includes Active/Active, circuit, BGP, and other design considerations
After my previous post about network edge design with dual internet providers (dual ISP or dual internet circuits) and ISP fail over I received a lot of questions afterwards regarding certain aspects of network edge design. If you haven't read the first detailed post please check it out here as that is somewhat of a primer for this post. The article included different edge designs from a branch office (single homed) with one provider and static routing to a larger enterprise with 2 providers using BGP (dual single-homed) to a provider edge point of presence (POP) example (dual multi-homed).
Although we'll go into more depth on some topics from part 1 of the network edge here, there are some important aspects that we will not be covering in this post, so it would be important to review the first network design post before proceeding here if you are new and being introduced to the topic and are here for initial research. Of course we can't cover everything but some of the topics discussed are directly related to questions or comments I received from readers.
I'd like to extrapolate some topics and also add additional color and information to help you set the network up to meet your goals. We'll look at BGP more and physical design of your circuits, however we aren't looking at like API end points or load balancers, storage containers etc, which you might find on the edge. We'll sprinkle some configuration in there but I prefer not to just rehash configuration guides you can easily find. Many vendors and companies have come out the last few years about "redefining the edge" and there is even micro data centers being built to help, however here we are focused on the on-prem networking aspects, not cloud, and not looking at things non-network engineers have created!
Topics covered in this post:
Circuit design
Active/Active considerations
Additional configurations and thoughts
Data Center Edge Design
DMZ redundancy without port-channels
Network edge design check list
Important Topics not covered in this post but are covered in part 1:
Business goals and budgeting (the FIRST step)
Initial design options and IP Addressing
High-level and Low-level network edge topologies
Route filtering and basic BGP configurations
Active/Passive edge routing
Edge traffic filtering with access-lists
Documentation in diagrams
Circuit Design
When designing a circuit, as we've talked about previously you want to consider the network paths both physical and logical your traffic will take which means equipment and location(s). You want to have redundant equipment and have diversity with things like fiber and geographical area. But how do you do that? What does this look like? Lets dive in.
Common Language Location Identifier code (CLLI code) is a common way in the United States you can use to identify Central Offices (CO, larger exchange/network interchange points) and Points of Presence (POP, smaller to medium exchange point) and serving wiring centers (SWC, i.e. cable plant buildings). These terms are widely used in the provider world and sometimes interchangeably these days. They are carry over terms from the old telephone days.
I'd be interested to know how other countries and places like Europe, or India and Australia do this. The reason the code is important is although you can have different addresses sometimes they can be the actual same location which the CLLI code can help identify; along with helping document places of interest in your design.
Consider when designing a circuit or having one built for you the path your traffic will take on your dual circuit edge network setup. Whether its for intra-network connectivity between POPs or you are setting up diverse internet paths for your network and you don't want to use the local data center's internet access it's important to trace out the path. Although there could be different addresses listed for the CO it could still be within the same center "campus", I use quotes here on campus because in a downtown metro area there could be different associated buildings on the same plot of land with 4 different addresses similar to a campus.
Having the CLLI code can help determine if its considered the same CO. Using google maps can help and I also found this site useful as well, if you don't have internal tools to do this.
Typically from a location there will be a local wiring center which is where a lot of the local fiber will terminate, its like the first stop in the journey before going to a larger central office or a place where peering takes place. It can include Layer 1, 2 or Layer 3 equipment but typically its for local connectivity and fiber splicing (all of these can be the same or separate depending on the city/country, local path, network and provider etc.).
Here is how I would recommend to look at diversity in the path and maybe some questions you should think about. I'm referencing fiber as that is the most reliable and highest performing medium, but could be replaced with your layer 1 medium of choice. We'll talk about some outside plant items and some logical IP items.
First from your location the diversity starts with equipment inside the building like customer premises equipment (CPE), then the fiber exits from the building to the street, and from there to the SWC. Then from the SWC you need to see where it goes for the central office or point of presence of upstream connectivity and do the same physical path tracing between those geographies (some of you might have nice internal tooling like Arcgis or Lightyear to do this). Typically you can request something like a KMZ file which will show the fiber path layered on a map.
In general it can be cost prohibitive to build multiple entries into a location or between locations, so sometimes compromises like using a common conduit will need to be made by you. Once the fiber gets onto the street though, is there different vaults for underground cable or different poles for aerial paths of the cable? Which streets do your 2 fibers travel on? Is this passive optical or dark fiber? Commonalities here should be avoided if possible. If both fiber are aerial it could be higher risk than if they are underground.
Then we arrive to (maybe) two different SWC that are across town from each other, this hopefully ensures you have physical local access or last-mile diversity once the physical fibers, paths, and SWC are confirmed in the topology. However keep in mind if they're in the same town they could be fed power from the same place by the same power companies!
Sometimes if the building is off-net for you or the provider, there will be a need to contract with the local exchange carrier (LEC) for network access, in those cases it might travel to a POP before a CO etc. where a Network to Network interconnect exists (NNI). Inside each of these locations there is fiber and equipment that could be common between circuits, so it's important in the design to utilize different switches and routers within your network and the provider network for NNI. Hostnames and CLLI codes again can help here. If there is only one NNI in that area for both circuits then you want to consider provider diversity. You should investigate your circuit more thoroughly if there is off-net connectivity involved.
From there you should analyze the middle-mile or intermediate path from the wiring center to the central office the same way you did the first leg of the path. Then once inside the egress point do you have different points of network terminations or interconnections with your neighbor's equipment? Think of diverse switches and routers again! As we mentioned in part 1 for the cross connections inside the data center, if you have two circuits you'll want separate cables if possible.
Here is an example of how things can look for a diverse dual edge network design from a high-level diagram perspective. The colocation is a smaller data center without much connectivity, but it shows things from a L1/L2 perspective. There are many tools like GIS that can show actual physical path tracing of the fiber on a map.
Additionally, you can see that although edge router 2 is connected to provider 2, the data center is off-net for them and requires them to use provider 1's L1/L2 connectivity to the site, however provider 1's connection for edge router 1 is using direct fiber, so we have physical and logical cabling diversity once leaving the colocation. The NNI is also at a different POP than what edge router 1 is peering to, and finally the ASN is diverse upstream for peering and edge connectivity for Layer 3 diversification.
At higher layer 3 levels you should consider the IP space and the BGP ASN paths your traffic will take. Is your internet access from two providers with different ASN or is it one provider with the same ASN? If its only 1 AS and there is a peering issue with them then that could affect your connectivity. Alternatively for enterprise folks, if you don't have provider independent IP space you might have to take that road of using the same provider. Whereas if you have your own IP space you can have multiple peerings with different entities for better routing protection. The example above would require provider independent space.
Keep in mind some providers will require you to order 2 circuits with them at the same time under the same order if you want them to be independently diverse, due to if there is maintenance in the future its possible that equipment could be consolidated or cables to be moved to the same card without some sort of initial order documenting it within the system.
Also even though you might be using two different carriers in your dual ISP network, its possible the last-mile is from only one provider who is the sole owner of facilities to the building thereby using the same switch or fiber cable (as shown in the above diagram). Local 'eyes-on' or documentation confirmation is very imperative here. Even with wave circuits you need to review your fiber paths for working and protection in order to facilitate proper resiliency so keep these items in mind.
One final thought would be if you are looking at different locations to host and setup a colo for edge routing then you probably want to look at the location's interconnection in the Peering DB for example to help in the decision. More interconnected is probably better and means there is probably lower pricing and various fiber options. Once you're there and want to pick a peer, something like the Hurricane Electric tool can help with researching that.
Active/Active Design
We talked about Active/Active setups in part 1 and I wanted to add some more thoughts surrounding this type of edge network intention.
As you can see from the previous section's diagram we listed different cities for our edge POPs with providers with different transport mediums, one is dark fiber and one is metro fiber type access. Although this shows optimal diversity there could be drastically different Service Level Agreements and performance with those or other types of examples which is one reason I chose those.
That should be something to contemplate when selecting access circuits. The metro fiber has relative middle mile protection whereas dark fiber probably does not. The on-net fiber probably provides better latency and jitter than an off-net circuit that switches between two provider networks and could even hair-pin to the NNI.
The packet delivery SLA for the on-net fiber might be 99.99% whereas the off-net circuit might only have a 99.9% packet delivery guarantee. A small difference but some people might care about it. Consider this for your traffic in Active/Active edge routing because if you have latency sensitive flows the customer experience could be different on a per-circuit-flow basis.
Obviously this isn't as much of an issue in the last 5 years as with the rise of SD-WAN along with equipment improvements, and the Internet's latency improving has allowed public network's to generally become more faster and predictable. But think about if a branch office has a DIA primary and a 4G or satellite backup, the VOIP experience could be drastically different when running A/A, therefore when there is a drastic difference between your two circuits of things like reliability, latency, jitter, it's probably not wise (even with SD-WAN) to run A/A or at least A/A for VOIP or other latency sensitive applications.
Another thing is with the selection of two cities is to analyze who you are serving from these boundaries or what you are accessing between locations as with different cities there could be different paths and therefore different latency between them. It doesn't matter if you have 10Gb circuits if the latency isn't optimal then performance can suffer. It's important to try and obtain metrics directly, otherwise you can use some public tools. To get an idea of a baseline of latency between cities some tools like Global ping or here is one from Uniti fiber as an example.
Be sure to research other features like BGP multipath for Active/Active designs that are multi-homed as you could potentially better utilize all of your active neighbors to justify return on monthly spend or improve capacity.
ASN Filtering
The question of running full tables or partial tables always comes up and the answer is inevitably "it depends". If this is an enterprise data center or something like a on-premises DC or 2 serving customers a single website or app then default-route only is probably fine as you're likely Active/Backup on the border. If you're a provider with multiple peers and serving many networks then probably full tables with Active/Active. Regardless of either framework be sure to only advertise your networks or your customer's network and not anything you shouldn't be! Strict routing policy here is paramount.
There are times where you might be Active/Backup where you could still be asking for full tables and use filtering to provide advantages. Something where you have some specific cloud destinations you upload traffic to or want to perform some one-way load balancing you could leak some prefixes from one of the routers to force the traffic to egress that direction. You'd add like a route-map entry for that prefix above your default-only allow with better local preference on the router you'd want the egress preferred. However if you were wanting to load-balance inbound traffic then full tables won't make a difference as obviously that doesn't influence inbound policy.
In the case of accepting only local routes from each ASN you are neighboring with, you could setup your routing as Active/Active and let inbound flows fall where they may as within those local provider's networks the traffic will be symmetrical. That means no pre-pending and additionally leaking some prefixes in from each provider into your edge routing table. Although eventually you will need to select where your default will go, this is where latency, bandwidth, and cost can come into play to help you select that path. You'll probably apply communities to help control traffic here too, we'll review that soon.
Additionally, even though you are filtering the majority of the routes, if you're accepting full internet routing tables you still need to ensure that your router's memory can handle that big of a RIB even if you are filtering them out and using the default-only.
Another advantage of receiving full tables without a default as opposed to a default-route only is so you can filter traffic based on the destination ASN or even ASNs in the AS-path, a geo-block type of setup or blocking adversarial networks during normal or hi-jack scenarios based on AS-filtering. By doing this without a default-route present you can send undesirable traffic to null. Another could be filtering out excessive AS pre-pending from your route table. This is shown in an example configuration below.
Cisco IOS-XR example
prefix-set Bogons-all
10.0.0.0/8 le 32,
100.64.0.0/10 le 32,
127.0.0.0/8 le 32,
169.254.0.0/16 le 32,
172.16.0.0/12 le 32,
192.0.2.0/24 le 32,
192.168.0.0/16 le 32,
end-set
!
as-path-set BGP_Excess_Prepend
ios-regex '+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_.+'
end-set
!
route-policy BGP_AS_Filter
if as-path originates-from '65008' or as-path originates-from '65009' then
drop
else
if as-path passes-through '65007' or as-path passes-through '65008' then
drop
else
if as-path in BGP_Excess_Prepend then
drop
else
if destination in Bogons-all then
drop
else
if as-path unique-length ge 12 then
drop
endif
pass
end
end
end-policyI tried a lot of different IOS-XR combos and testing to filter out excessive ASN pre-pending and also searched google/AI but it seems I couldn't get it working as exactly as I intended, some examples were '([0-9]+)_\\1{5,}' or '_([0-9]+)_\\1_\\1_\\1_\\1_' or '([0-9]+)(_\1)(_\1)(_\1)(_\1)(_\1)(_\1)(_\1)' etc. etc. All AI provided examples failed.
However the as-path-set BGP_Excess_Prepend shown above will block excessive pre-pended routes which the as-path unique-length knob cannot block. See output below.
After filtering, note that AS 65007 is gone because it's pre-pended too much.
With IOS-XE you have to use a path ACL which is slightly different, showing a wider net in this example.
IOS-XE example
ip as-path access-list 1 permit _([0-9]+)_\1_\1_\1_\1_\1_\1
ip as-path access-list 1 deny .*
!
route-map BGP-AS-Filter deny 10
match as-path 1
!
route-map BGP-AS-Filter permit 10
!
Oh no just blackholed friendly AS 65005, definitely be careful! Purposely done to show how things can go wrong. Analyze your routing table for long AS paths to see if you have some candidate prefixes you'd want to remove.
Similar Juniper Example
policy-options {
prefix-list Bogons-all {
10.0.0.0/8;
100.64.0.0/10;
127.0.0.0/8;
169.254.0.0/16;
172.16.0.0/12;
192.0.2.0/24;
192.168.0.0/16;
}
policy-statement BGP_AS_Filter {
term Drop-src-65009 {
from {
as-path "65009.*";
}
then reject;
}
term Drop-src-65006 {
from {
as-path "65006.*";
}
then reject;
}
term Drop-transit-65009 {
from {
as-path ".*65009.*";
}
then reject;
}
term Drop-transit-65006 {
from {
as-path ".*65006.*";
}
then reject;
}
term Drop-Private-ASN {
from {
as-path ".* (64512-65535) .*";
}
then reject;
}
term Allow-all {
then accept;
}
}
}
Communities and Templates
Another item that pops up frequently that was only mentioned briefly in part 1 was regarding tagging BGP communities. Its generally best practice to define a few different policies in your network so you can tag different types of traffic using the BGP community attribute to create strategies with. Things like customer received routes, transit routes, internal announcements, geographies and so forth are good examples to help with traffic engineering and tactically deciding how the network can respond. Usually you'll want to strip external communities inbound and outbound except for a few scenarios which we will cover. Local preference is the primary BGP mechanism we'll want to use.
You want to tag or assign routes at their origin along with inbound at the border so you can better determine policy internally where you would match communities downstream inside your AS. Sometimes you could be troubleshooting and it's easy to glean information just by looking at what communities you have assigned to the route. The next diagram shows the basic principle.
Setting communities at all points can allow routing policy to determine the path to prefer private, then internet exchange (IX), and then here we have aggregate advertisements to route to use transit (most expensive) as a last resort. For inbound advertisements question if the same network (network 1A in the diagram) is sending specific routes from a private peering and also from an IX and you want to ensure all routers in your network to use the private peering connection instead of the IX due to bandwidth efficiency or ROI and so forth.
Same as if you were Network 1A you might want to tag with the neighboring networks communities to dictate policy for your ingress from them if it was supported. You want to reject other unknown communities and strip your own outbound to avoid pollution or false manipulation inside your network or other's jurisdiction. In addition, if you are telling other networks to use certain communities to set local preference within your AS you might want to have some different options internally and also give yourself the option to override that LP if that was necessary.
Cisco IOS-XR Example
community-set community-110
65001:1234
end-set
!
community-set community-90
65001:5678
end-set
!
route-policy Set-local-pref
if community matches-any community-110 then
set local-preference 110
else
if community matches-any community-90 then
set local-preference 90
endif
pass
end
end-policy
!
route-policy Tag-inbound-routes
delete community all
set community community-110 additive
pass
end-policyAgain you'll want to ensure all routers treat that community the same with a certain local preference via routing policy, in this case preferring it. XR is very granular and you could nest these policies inside of other route-policies so the possibilities are endless and with communities you can have a lot of options and can signal things like no-export and DDoS mitigation via RTBH.
Here is an example of communities that Lumen uses who is a large Tier 1 or that which customers or neighbors can use.
BGP templates or peer groups offer a repeatable way to have standardized settings for neighbors which is meaningful for current or future automation plans you might have. You can apply common settings like route-maps or policy, which would contain tagging of communities or filtering, including BGP timers and descriptions or max prefixes for instance. You can create different templates by peer type like we thought about with varying types of communities. By having a template it becomes easier to automate turn-ups with customers with scripts or automated pushes via tickets and ensures uniformity across your autonomous system.
Other Thoughts
I still see a lot of people using and recommending things like IP SLA, object tracking, conditional advertisements or policy-based routing at the border and I want to say really these are not needed if you are doing things right with BGP. The only time I think mechanisms like that are to be used are for branch offices that have a small PA IP addresses assigned and aren't using or support BGP. Furthermore, if you are using ping monitors always try to use them against destinations you control and if you control it then use TWAMP or similiar.
If you want better detection then use BFD or more aggressive BGP timers and design it right. Don't use these other non-protocol mechanisms to track routes or availability because you are just going to end up with flapping routes and phantom issues which require more attention and cause your reliability to suffer. There are even some built into BGP like next-hop address tracking which can also cause excessive dampening.
Likewise a question about full tables on MPLS PE routers that are inside the network closer to the core and behind edge routers (usually connecting CEs) I want to provide an opinion on. If the network is small with perhaps 2 edge routers, 2 PE routers and 2 P routers symmetrically setup similar to the communities diagram above, then full tables probably aren't needed for simplicity sake. However as the network becomes larger it becomes a necessity to run full tables on the PE routers. For example you can run an internet VRF and carry traffic for DIA customers attached to the core PEs and import them if needed, so you'd be running labels throughout and not a separate pure IP network for DIA full-table customers and MPLS for VPN customers. This can allow you to keep a BGP free core and still have a more simplified route leaking type setup than needing to run something like endless layer 2 xconnects to achieve the same goal for those customers. Plus if there is large public routing changes your core LSR aren't affected. Hardware can be a limitation though.
Moreover, if running a default only for a handful of perimeter routers and PE routers you are more likely to encounter excessive asymmetric traffic flows because the traffic will be load balancing inefficiently and returning incorrectly from ER to PE. Conversely, if you are running full tables between your A/A border and internal PE you are clearly going to see more balanced return traffic and symmetrical flows to and from the core (that's better if firewalls are involved).
Data Center Edge Design
I see a lot of posts consistently asking about connecting firewalls and edge routers in the data center within a Clos spine/leaf topology. Its very simple really in my opinion and that is to use a dedicated set of "service" leafs for this purpose. I wouldn't necessarily say a second pair of dedicated switches in addition to those service leafs is mandatory for the DMZ though like with other networks we've discussed but it always depends.
The service leafs terminate specialty connections that are different than the standardized top-of-rack switches connected to VM host servers. These connections would be the DCI layer 3 circuits, the boundary router DMZ connections, the DC firewalls, and other non-leaf switches used for OOB management. Something like cross connects where you might have only a handful and couldn't symmetrically fit in an odd numbered spines to leaf ratio (e.g. due to cost or acceptable amount of bandwidth ratios).
Having these devices connected to the service leafs offers predictability and centralized control and configuration which are things we often look for in a good design. Having all leafs connect to all spines means ECMP routing and bandwidth is the same for all leaf switches versus having things plugged into certain spines. This keeps its simple and doesn't violate Clos principles which is what we want!
I show in the above high-level diagram in a case of how things can be imagined in the data center. You probably notice a lot of it looks similar to other DMZ diagrams from part 1 and shown in this post.
I list that VPC/MLAG (or even a chassis possibly) is preferred for the service leafs in order to facilitate LACP port-channels for redundancy. Potentially for larger networks and heavily automated networks this might not be as ideal since its different than the TOR standardized switch but your mileage may vary. You could easily accomplish the same amount of redundancy and configuration without that and have the service leafs connect directly but you'd also be restricted as you couldn't use port-channels (particularly covered in the next section), Chassis is another solution. HA Firewalls is easy for the stacking type setup and can work without it and firewall clustering is probably better for non-vpc high bandwidth networks.
Most major firewalls these days (or at least only the ones you should be using) even support EVPN and Vxlan (i.e. VTEP directly on the firewalls), hence you could technically have them participate in the fabric directly; or just have them as layer 2 (i.e. VTEP on service leaf switches not firewalls) depending on how you'd want it to connect or based on hardware capabilities.
Consider port density as well, here only 1 link per spine is shown but recent developments with AI bandwidth needs are calling for additional connections per-spine and switches are coming with 400Gb ports which means ports at the spine layer are becoming more premium (disregarding breakout options). Also at the spine layer I don't generally recommend specialty connections there as its similar to the core layer where we don't want complex manipulation or configurations just speed. That, along with avoiding a divergence on the typical way to think of how the traffic flows are in standard Clos style networks.
Traffic patterns could be fairly predictable with even amounts of available bandwidth but having appropriately sized firewalls and routers is essential in the DC because you'd probably have 100-400Gb connections between spine and leaf but then you'd be stepping down which can cause buffer issues and hardware bottlenecks if you have demanding applications. Therefore you'd want to have higher sized interfaces on the firewalls and routers like 25/100Gb if possible in high-demand environments.
Lastly, notice the green and red lines from the firewalls visualizing how the traffic travels from inside to outside which applies to intra-DC VM to VM traffic on different vlans/vrfs or VMs to the outside. All gateways are on the firewalls which controls all intra-DC and inside to outside traffic and so forth, when when a VM needs to leave it local network traffic will travel to the firwalls before heading to naother network our beyond the edge.
Edge Redundancy without Port-Channels
Often in the DMZ there is a need for Ethernet switches, either a dedicated pair or one pair that is doing double duty. These are the switches between the border routers and firewalls or other routers that help bridge the gap with their port density.
You can see from the diagram below we show port-channels or link-aggregation between the routers and firewalls to the switches which are joined in a "VPC" stacked-like pair which builds them as one switch. Ideally we want a chassis, switch stack, or VPC/MLAG type setup which allows us to create port-channels between the devices. This means there is less impact for hardware failures. However what if the switches are separate and we cannot do that? This is often the case for a lot of people. We can get a little creative if necessary but in a standardized fashion.
To accomplish an analogous setup to get the redundancy of having two switches and two connections each we can take a classic config from the service provider access network of bridged interfaces.
These allow you to create a logical interface which you'd connect to multiple switch ports to allow your interfaces to participate in layer 2 just like connecting a switch with a SVI interface (which you can do if using like an Arista switch or other types of linux network devices for border routers). These are typically used to connect multiple customers at the access level for certain types of ISP networks but will allow us to get resiliency resembling a port-channel here.
We'll first enable spanning-tree on the routers so they participate in STP with the switches, so one port will be active on switch 1 for example and then for switch 2 it will be in a blocking state available for backup. We have to create a bridge interface for untagged vlan 1 as well so BPDUs can be exchanged.
If you are not using SVI on the firewall and/or it doesn't support this specific configuration then we would remove one of the firewall interfaces to each switch and utilize the high availability mechanisms to fail over to firewall 2 in the case of switch 1 failing. That in itself is disruptive but the firewall type and HA config will determine by how much. The switches are setup with standard trunk aka tagged vlan type ports.
Using the next figure. if SWT1 would to be offline then FRW2 will take over as primary and RTR1 will switch over to its' G0/0/2 interface connected to SWT2 keeping data flowing through the DMZ the same way and be less disruptive on the public routing side since RTR1 is still advertising prefixes for inbound traffic. Although not as clean as with a chassis and port-channels and more disruptive internally during firewall failure scenarios, we still achieves higher resiliency for our perimeter.
Finally, if you wanted it even simpler then you'd eliminate the connection from RTR1 to SWT2 and RTR2 to SWT1 and just have a ring setup, however if SWT1 is down then traffic would flow to RTR2 which is still connected to the back end network and then flow to RTR1 via iBGP because it still is the preferred path upstream for inbound & outbound, this is when someone might try to use IP SLA and I will say again don't do it. Alternatively, you can just do away with all of that and run layer 3 direct from router/switch/firewall and separate it via VRFs which might be simpler and cleaner to some.
RTR1 Cisco IOS Example
spanning-tree mode rapid-pvst
!
bridge irb
bridge-domain 1
bridge-domain 105
bridge-domain 106
!
interface GigabitEthernet0/0/2
description to SWT1
service instance 1 ethernet
encapsulation untagged
bridge-domain 1
!
service instance 105
description SI-inside-dmz-private
encapsulation dot1q 105
rewrite ingress tag pop 1 symmetric
bridge-domain 105
!
service instance 106
description SI-inside-dmz-swt1-swt2
encapsulation dot1q 106
rewrite ingress tag pop 1 symmetric
bridge-domain 106
!
interface GigabitEthernet0/0/3
description to SWT2
service instance 1 ethernet
encapsulation untagged
bridge-domain 1
!
service instance 105
description SI-inside-dmz-private
encapsulation dot1q 105
rewrite ingress tag pop 1 symmetric
bridge-domain 105
!
service instance 106
description SI-inside-dmz-public
encapsulation dot1q 106
rewrite ingress tag pop 1 symmetric
bridge-domain 106
!
interface BDI105
description BDI-inside-dmz-private
ip address 10.1.1.98 255.255.255.248
no shut
interface BDI106
description BDI-inside-dmz-public
ip address 3.3.3.193 255.255.255.248
no shut
!
end
RTR1 Similar Juniper Example (Juniper has more ways to do this it seems)
set interfaces ge-0/0/2 encapsulation flexible-ethernet-services
set interfaces ge-0/0/2 flexible-vlan-tagging
set interfaces ge-0/0/2 description "to SWT1"
set interfaces ge-0/0/2 unit 1 encapsulation vlan-bridge
set interfaces ge-0/0/2 unit 1 vlan-id 1
set interfaces ge-0/0/2 unit 105 encapsulation vlan-bridge
set interfaces ge-0/0/2 unit 105 vlan-id 105
set interfaces ge-0/0/2 unit 105 input-vlan-map pop
set interfaces ge-0/0/2 unit 105 output-vlan-map push
set interfaces ge-0/0/2 unit 106 encapsulation vlan-bridge
set interfaces ge-0/0/2 unit 106 vlan-id 106
set interfaces ge-0/0/2 unit 106 input-vlan-map pop
set interfaces ge-0/0/2 unit 106 output-vlan-map push
!
set interfaces ge-0/0/3 encapsulation flexible-ethernet-services
set interfaces ge-0/0/3 flexible-vlan-tagging
set interfaces ge-0/0/3 description "to SWT2"
set interfaces ge-0/0/3 unit 1 encapsulation vlan-bridge
set interfaces ge-0/0/3 unit 1 vlan-id 1
set interfaces ge-0/0/3 unit 105 encapsulation vlan-bridge
set interfaces ge-0/0/3 unit 105 vlan-id 105
set interfaces ge-0/0/3 unit 105 input-vlan-map pop
set interfaces ge-0/0/3 unit 105 output-vlan-map push
set interfaces ge-0/0/3 unit 106 encapsulation vlan-bridge
set interfaces ge-0/0/3 unit 106 vlan-id 106
set interfaces ge-0/0/3 unit 106 input-vlan-map pop
set interfaces ge-0/0/3 unit 106 output-vlan-map push
!
set vlans VLAN1 vlan-id 1
set vlans VLAN105 vlan-id 105
set vlans VLAN106 vlan-id 106
!
set bridge-domains VLAN1 domain-type bridge
set bridge-domains VLAN1 vlan-id 1
set bridge-domains VLAN1 interface ge-0/0/2.1
set bridge-domains VLAN1 interface ge-0/0/3.1
!
set bridge-domains VLAN105 domain-type bridge
set bridge-domains VLAN105 vlan-id 105
set bridge-domains VLAN105 interface ge-0/0/2.105
set bridge-domains VLAN105 interface ge-0/0/3.105
set bridge-domains VLAN105 routing-interface irb.105
!
set bridge-domains VLAN106 domain-type bridge
set bridge-domains VLAN106 vlan-id 106
set bridge-domains VLAN106 interface ge-0/0/2.106
set bridge-domains VLAN106 interface ge-0/0/3.106
set bridge-domains VLAN106 routing-interface irb.106
!
set interfaces irb unit 105 family inet address 10.1.1.98/29
set interfaces irb unit 105 description "BDI-inside-dmz-private"
set interfaces irb unit 106 family inet address 3.3.3.193/29
set interfaces irb unit 106 description "BDI-inside-dmz-public"
!
set protocols rstp bridge-priority 8k
set interfaces ge-0/0/2 rstp
set interfaces ge-0/0/3 rstp
!
commit
Network Edge Design Checklist
Lets list out all of the elements we might need for a project when doing a proper network edge design. Obviously we go for a more comprehensive list to then remove what isn't needed and keep what is needed.
Administration
Business -
Goals
Justification
Monthly reoccurring cost budget (or profit margin)
Non-reoccurring costs
Architecture diagram
Hardware -
Budget
Vendor Make/Model
Feature support
Licenses/Support
Future service needs
End-to-End design diagram
Configuration -
Existing standards
Anything unique to take into account
Preparatory work
IRR, RPKI, RIR administration
Contracts, LOAs, and Service level agreements (Customer, Peer, or Provider)
Project timeline planning
Construction planning (many items related to this line)
Maintenance/life cycle planning
Circuits
Transport mediums available
Location and provider options
SWC
POP
CO
NNI for off-net
Diversity
Physical cable and path from inside premises to last-mile wiring center
Intermediate physical cable and path if applicable to middle-mile POP
Physical cable and path from wiring center (or middle-mile facility) to CO
Redundant paths or protection (layer 1/optical, Layer 2 and/or layer 3)
Optics/hand offs interoperability
CPE and PE equipment at each location (distinct in # 3,4,5 above)
Latency, Jitter, Bandwidth, SLA
Scheduling
Layer 2
Router to Router connections
Switch to Switch connections
Paths between Layer 3 points/interconnects
Additional configuration items -
VLANs (S/C)
Port-Channel
Stacking/VPC/MLAG
Cross-connects
Layer 3
IP addressing
BGP -
ASN
Ingress prefix/route acceptance
Egress prefix advertisements
Route filtering
Local Preference
Communities (Assigning, stripping, and/or Matching)
VRFs (E.g. Lite, or for L3VPNs)
Upstream peering beyond neighbor
Active/Active or Active/Passive routing considerations
Ingress traffic filtering (ACL)
QoS -
Total bandwidth CIR
Shaping/Policing (match CIR)
Queues (by DSCP if applicable)
Firewall (certain cases) -
Zones
NAT pools
Configuration feature support
HA fail over
Inbound/Outbound routing from edge to backbone or internal network -
IGP
Communities/Local preference again
P/PE routers
Hot/Cold Potato
Documentation
Circuit IDs
Support & contact information
Billing/account numbers (got to pay or collect those bills)
Physical Pictures of setup
High-level/Low-Level diagrams
Layer 2
Layer 3
As-builts
Cable path (map of splice points, aerial/UG, pedestals, vaults etc.)
Rack Elevations
Documentation from Peer/Provider
Configuration Documentation -
Templates (e.g. Automation)
Route-maps
IP Addressing
ACLs
Spreadsheet or software tool updating
Source of truth data inputs
PSA: RPKI > IRR
Resource Public Key Infrastructure should be on everyone's roadmaps (it's on mine) if its not already because the Internet Routing Register is not as secure for origin validation versus RPKI. This is because IRR is known to have a lot of stale entries and could be abused more easily versus RPKI, but as you might know RPKI adoption has been slow due to it's complexities. MANRS posted about inconsistent or suspicious IRR entries last year.
An IRR entry is a good first step to matching your BGP ASN with your prefixes and if you haven't done that please do. I know there is still a lot of who haven't as I seen it when browsing BGP checkers, but please also look into RPKI as it's a more secure way to prevent BGP prefix hi-jacking. You can also use the building blocks of this post like with the ASN filtering and apply it to RPKI. I haven't done this in production, so I probably won't have further detail on this on my blog right now as I usually don't post anything that I haven't done or seen in production myself; but point is it's essential for the security of the internet. Links to follow:
North America and Caribbean
Africa
Asia Pacific
Latin and South America
Europe and Middle East
Conclusion
We finally have reached and end to part 2 of network edge design. We dove deeper on some topics from part 1 and discussed the important elements of active/active setups and had examples of BGP filtering configurations. We looked at some possible configurations and topologies of not being able to use port-channels or how the edge in the data center can be imagined, as those are questions I have been asked along with seeing on social media. We still haven't looked at every BGP possibility, redundant data centers, or items like QoS so perhaps we can do a part 3 if there is interest.
It's vital to remember that as great as the internet is there are still security concerns which should be top of mind for you, while still taking transport items like latency and circuit design into your edge routing project. The network edge serves as the entry and exit point of your network to the world and I hope this post's information and the final check list helps you from any pitfalls you could encounter. Thanks for reading.
Would you like to know more?
As always this information is provided on a best effort, as-is basis and I take no responsibility or liability to any networks, businesses, entities, or operators who use this information. Always research and develop your own plan and configuration.
