What is AWS Direct Connect?

AWS Direct Connect establishes a direct private connection from your equipment to AWS. Use AWS Direct Connect to form a dedicated network between your physical hardware (e.g., colocation environment, office, etc.) and AWS resources.

AWS connections use 802.1q VLANs, which is the industry standard. The benefit to this is that the connection can be partitioned into multiple private and public virtual interfaces. This means that an organization can use a single connection to access private resources, such as Amazon EC2, as well as access an Amazon S3 object over a public environment. AWS Direct Connect maintains network separation between public and private connections at all times.

For added network flexibility, you can edit the virtual interfaces at any time.

How to configure AWS Direct Connect.

Benefits of Amazon Web Services Direct Connect

Besides an increase in data throughput, in many use cases, AWS Direct Connect can reduce your network costs. AWS also provides a consistently high-quality network that is a better experience than an Internet-based connection.

Flexible

Signing up for the service is simple; everything is performed via the AWS Management Console. The Management Console is a single access point for managing all AWS virtual interfaces and connections. Furthermore, after configuring at least one virtual interface, there are customized router templates available for download for diverse networking equipment.

AWS Direct Connect provides secure network scaling for every need. It can provide 1 Gbps and 10 Gbps connections, and it makes provisioning multiple connections easy. Moreover, instead of accessing your instance of Amazon VPC over an Internet-based VPN connection, you can use AWS Direct Connect. Considering VPN hardware often doesn’t support data connections above 4 Gbps, AWS can significantly improve your connectivity.

Consistent Connectivity

Connectivity over the Internet may fluctuate, as you do not have full control over how data gets from start to finish.

With AWS Direct Connect, an organization can choose which data is routed in which way, so you have more control over the connection. This kind of dedicated network may offer a more consistent flow of data than an Internet-based network. Also, there are no data limits you can transfer using AWS Direct Connect.

Compatibility with Amazon Web Services

Using AWS Direct Connect, you can establish connections to your private and public AWS resources in a given AWS region. This way, an organization can transfer data from and to AWS, and bypass Internet service providers and any possible network instability.

Maintain a Dedicated Network with Amazon’s Cloud Services

As already mentioned, AWS Direct Connect can serve as a replacement for a VPN hardware connection to your Amazon VPC. This type of connection is entirely private, and if utilizing several virtual interfaces, you can establish links to several distinct instances of Amazon VPC. Complete network isolation is guaranteed.

How to Configure AWS Direct Connect

Choose an AWS Edge Location

To maintain minimum network latency, Amazon Web Services are offered through AWS edge locations. A regional endpoint is a URL that serves as an entry point for Amazon’s web services. Amazon Web Locations can be checked at https://aws.amazon.com/directconnect/details/.

Customers are encouraged to access AWS Direct Connect through an AWS Direct Location. By collocating your equipment at a designated edge location, you can use the existing network circuits between a data center and an AWS device.

This connection delivers up to 10 Gbps port speeds.

Work with an AWS Partner Network or Network Provider

An alternative solution would be to work with a partner in the AWS Network Partner (APN) or a network provider to connect your on-premise or colocation router to an AWS Direct Connect location.

This connection also provides 1 Gbps or higher port speeds.

Hosted Connection

Another option is to hire a partner in the AWS Partner Network (APN) to create a hosted connection for your organization. If you opt for this solution, after signing up for an AWS Direct Connect account you need to accept this connection and then create a virtual interface.

This type of connection delivers slower port speeds at sub-1 Gbps and supports only a single virtual interface.

Account, Connection, and Virtual Interface

After deciding on an AWS location and type of connection, sign up for AWS Direct Connect and then create an AWS Direct Connect connection, download the LOA-CFA and create a virtual interface.

This article will guide you through each step.

Sign Up for AWS

Sign up for AWS Direct Connect by creating an AWS account at https://aws.amazon.com/. Follow the on-screen instructions and be prepared to receive a phone call from Amazon. You will need to enter a PIN using the phone keypad.

AWS Direct Connect Connection Request

Access the AWS Direct Connect console at https://console.aws.amazon.com/directconnect/. Select an AWS region and the required port speed. Port speed cannot be modified after the connection has been created.

Create an AWS Direct Connection

Use the AWS Direct Connect management console to create an AWS Direct Connection. Navigate to Connections and opt to create a Create Connection. A new dialog box displays the necessary fields.

create a connection with AWSAfter you have created a connection, you should receive a confirmation message as seen in the image below.

successful connection madeThe connection is in a “requested” state. The AWS Direct Connect staff is reviewing your request and will then supply a letter of authorization. Once available, you need to download LOA and send it to your network provider who is establishing the connection for you.

It may take up to three (3) business days to process the request.

Info
Common Name or FQDN
Organization Name (e.g., company)
Organizational Unit Name
Locality Name (e.g., city)
State/Region/Province (full name)
Country Code (2 letter code)
Email Address
Public Key
Description
FQDN is the fully qualified domain name of your website. It must be the same as what users type in the web browser.
The full legal name of your organization, including suffixes such as LLC, Corp, etc.
The division in your organization that deals with this certificate.
The city in which your organization is located.
The state or region in which your organization is located.
The country in which your organization is located. Always entered as a two-letter ISO code.
Email address used to contact the site’s webmaster.
An automatically-created key that’s generated with the CSR and goes into the certificate.
Example
www.phoenixnap.com
PhoenixNAP, LLC
NOC
Phoenix
Arizona
U.S.
sitewebmaster@phoenixnap.com
An encoded text block similar to the private key. See an example of a private key below.

Download the Letter of Authorization and Connecting Facility Assignment (LOA-CFA)

After filling a request for a connection, AWS will process the application. It may take up to 72-hours for Amazon to review the request and provision a connection port. Amazon may request additional information via email. Respond within seven business day or the connection will be terminated.

Once the request has been accepted, download the Letter of Authorization and Connecting Facility Assignment. Simply put, this is Amazon giving you permission to establish and use the connection.

To download the LOA-CFA, log into your AWS Direct Connect account, navigate to Connections, and select the newly created connection. Choose Actions > Download LOA-CFA.

NOTE

If the link is unavailable, it means the letter of authorization is still not available. Check your email. If 72- hours have passed and you still haven’t received an email, contact AWS support.

An optional step is to enter the name of your network provider. It will appear with your organization’s name as the requester of the LOA-CFA. Download the letter of authorization. It will be downloaded as a PDF file.

Requesting Cross-Connect

After you have downloaded your letter of authorization, request a cross-connect connection. If you have equipment at the AWS Direct Connect location, contact your designated provider to establish a cross-connect connection. For example, if you have equipment at PhoenixNAP, Phoenix, you would send an email to sales@phoenixnap.com.

For a comprehensive list of AWS providers, refer to Amazon AWS documentation.

Cross-connect must be established within 90 days of granting the LOA-CFA. After 90 days, the letter of authorization expires. If the LOA-CFA expires, download it again from the AWS Direct Connect console and resend this to your network provider.

Hosted Connections

For speeds less than 1 Gbps, you cannot use the AWS console to request a connection. Instead, hire an AWS Direct Connect partner to create a hosted connection for you.

Accept a Hosted Connection

If an AWS partner creates a hosted connection for you, you only need to accept the connection after creating an AWS account.

Log into your AWS account at https://console.aws.amazon.com/directconnect/ and select the region in which the connection is located. Choose Connections, find the hosted connection, and select it.

logging into Amazon Web Services Account

Accept the connection to activate it. After activating your connection, the next step would be to create a virtual interface.

Virtual Interface

Once your connection’s state goes from “requested” to “available, you can create a virtual interface. Virtual interfaces are a prerequisite before using AWS Direct Connect. Bear in mind that you can create multiple virtual interfaces on a single AWS connection.

First, you need to be aware of the two types of virtual interfaces. Namely, there are public virtual interfaces, which are used to connect to public AWS resources. And then, there are private virtual interfaces which are used to connect to your instance of Amazon VPC. If an organization wants to communicate with several VPC instances, it should utilize a single virtual interface per VPC.

Before you establish a virtual interface, make sure you have the necessary information. Also, take into account that sub-1G connections are limited to a single virtual interface.

How to Create a Public Virtual Interface

If you are connecting to AWS public resources, perform the following steps.

NOTE

This type of virtual interface is not intended for use with Amazon VPC. For such connections, create a private virtual interface.

  1. Log into your AWS account at https://console.aws.amazon.com/directconnect/.
  2. Navigate to Connections, select the connection you intend to use and select Actions > Create Virtual Interface.
  3. Make sure to select Public as the appropriate option for your virtual interface.

create a Public Virtual Interface

In the Define Your New Public Virtual Interface, provide the following information and select Continue:

Field
Connection
Virtual Interface Name
Virtual Interface Owner
VLAN
(i.e., virtual local area network)
Address Family
Your router peer IP / Amazon router peer IP
BGP ASN
Prefixes you want to advertise
Description
Select an existing physical connection on which you wish to create a virtual interface.
Enter a name for your virtual interface.
Select My AWS Account if the virtual interface is to be used by you.
The ID number VLAN. The value must be between 1 and 4094. VLAN is required for data transfer in the AWS Direct Connect network.
IPv4 or IPv6. If you opt to configure an IPv6 BGB peer, select IPv6. The IPv6 addresses are automatically assigned from the AWS pool of IPv6 addresses. IPv6 addresses cannot be custom specified.
Enter two public IP addresses that your organization owns. If you do not have these, contact your network provider or AWS to provision the IP addresses.
Enter your gateway’s Border Gateway Protocol (BGP) Autonomous System Number (ASN).

If you are using a private ASN, it needs to be between 64512 and 65535.
The IPv4 CIDR destination addresses to which data should be routed over the virtual interface. Separate multiple addresses by commas.

Amazon verifies network ownership before approving any public virtual interface.
Example
PhoenixNAP Connection
PhoenixNAP Virtual Interface
My AWS Account
200
IPv4
7.5.3.5/37
65000
 

How to Create a Private Virtual Interface

If connecting to phoenixNAP’s AWS Direct Connect endpoint, you will need to configure all virtual interface options except for the VLAN (i.e., virtual local area network) field. PhoenixNAP provides the VLAN Number. This number will be between 1 and 4094, and it must comply with the Ethernet 802.1Q connection standard.

To create a private virtual interface, you need a public or private ASN and the VPC virtual private gateway (VPG) ID.

To begin the process of creating a private virtual interface:

  1. Navigate to https://console.aws.amazon.com/directconnect/and log into your AWS account.
  2. Choose Connections, select the connection to use and select Actions > Create Virtual Interface.
  3. Select the appropriate Virtual Interface type. In this case, click Private.

Field
Connection
Virtual Interface Name
Virtual Interface Owner
Connection To
VLAN
(i.e., virtual local area network)

Address Family

BGP ASN
Description
Select an existing physical connection on which you wish to create a virtual interface.
Enter a name for your virtual interface.
Select My AWS Account if the virtual interface is to be used by you.
If you do not already have a Direct Connect gateway, there will be an option to create one on the Create a Virtual Interface screen. You can create a DCG in any supported public region. If you have an existing configuration, please choose the DCG you have already configured.
The ID number VLAN. The value must be between 1, and 4094 and it mustn’t already be in use in your office. Avoid using 1, as this is typically used by management. VLAN is required for data transfer in the AWS Direct Connect network.

If collocating, your provider will supply the VLAN.
IPv4 or IPv6. If you opt to configure an IPv6 BGB peer, select IPv6. The IPv6 addresses are automatically assigned from the AWS pool of IPv6 addresses. IPv6 addresses cannot be custom specified.

On the other hand, you can specify IPv4 addresses.
Enter your gateway’s Border Gateway Protocol (BGP) Autonomous System Number (ASN).

Check the Auto-generate BGP key checkbox to have AWS generate a BGP MD-5 key.
Example
PhoenixNAP Connection
PhoenixNAP Virtual Interface
My AWS Account
 
200
IPv4
65000

Upon creation, the virtual interface will be in the state of “pending.”

NOTE

If you used the VPC wizard to create a virtual private center, route propagation should be automatically enabled. Routes will be automatically propagated to route tables. If you want to disable route propagation, you will need to do so manually.

Direct Connect Gateways

Direct Connect gateways can group private virtual interfaces and virtual private gateways that belong to a single AWS account. Use Direct Connect gateways to connect your AWS Direct Connect connection to a VPC in the same or different region. You do so by associating the Direct Connect gateway with the virtual private gateway of a VPC.

To create a Direct Connect Gateway:

  1. Log into your AWS account at https://console.aws.amazon.com/directconnect/.
  2. Select Direct Connect Gateways > Create Direct Connect Gateway.
  3. Provide the necessary information.

Info
Name
Amazon side ASN
Description
A descriptive name that will help you identify the AWS gateway.
Provide the ASN for the AWS side of the BGP session. For 16-bit ASN, the value must be between 64,512 to 65,534. For 32-bit ASN, the value must range from 4,200,000,000 to 4,294,967,294.
Example
Gateway for PhoenixNAP AWS
65000

AWS Direct Connect gateways have certain limitations

  • Multiple VPCs associated with a single Direct Connect gateway cannot communicate directly.
  • Multiple virtual interfaces that are associated with a single Direct Connect gateway cannot communicate directly.
  • A virtual interface associated with a Direct Connect gateway and a virtual private gateway associated with that same Direct Connect gateway cannot communicate directly.
  • A virtual private gateway may only be associated with a single Direct Connect gateway.
  • A virtual private gateway associated with a Direct Connect gateway must be attached to a VPC.
  • Currently, Direct Connect gateway cannot be used to connect to a VPC in the China region.

Create a Virtual Private Gateway in VPC - AWS Settings

Create a virtual private gateway and attach it to the VPC that contains the EC2 VMs you are trying to connect to. To create a VPG and attach it to a VPC:

  1. Log into your AWS account and select Virtual Private Gateways > Create Virtual Private Gateway.
  2. Enter a name for your VPG which will create a tag containing a key of Name and the value you have entered. If you intend to use the default AWS ASN, don’t change the ASN default selection. To type in a value, select Custom ASN and enter a value. It should be between 64512 and 65534 or 4200000000 and 4294967294.
  3. Select Create Virtual Private Gateway.
  4. Select the newly created VPG. Click Actions > Attach to VPC.
  5. Select the desired VPC and click Yes, Attach.

Associate the Virtual Private Gateway with an AWS Direct Connect Gateway

Associate the new VPG with the DCG you created earlier. To do so, you need to be in the same region in which the virtual private gateway is located. The same applies to disassociating VPGs. The VPG must be attached to a VPC.

  1. Log in to your AWS Direct Connect console at https://console.aws.amazon.com/directconnect/.
  2. Select the region in which your VPG is located.
  3. Select the Direct Connect Gateway drop-down, and click your desired Direct Connect gateway.
  4. ClickActions> Associate Virtual Private Gateway.
  5. Find and select the desired virtual private gateway, and select Associate.

If you want to check all your virtual private gateways in all regions associated with a single Direct Connect gateway, select Virtual Gateway Associations. This will list any existing associations.

Connecting a Bare Metal Backend to AWS via Direct Connect

NOTE

The configuration outlined below is an example of how a phoenixNAP customer would typically connect their Bare Metal backend to AWS Direct Connect. This may or may not apply to your use case.

Router Server

Choose a server from your PNAP Bare Metal inventory to become your Router Server.

IP Forwarding

Enable IP & IPv6 forwarding on chosen Router Server. Create a new file and name it /etc/sysctl.d/90-routing-sysctl.conf. The file should contain the following content:

# Sysctl for routing
#
# Routing: We need to forward packets
net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv4.conf.enp1s0f0.send_redirects=0

Load the New sysctl.conf File

Run the following command to load the newly created sysctl.conffile:

sudo sysctl -p /etc/sysctl.d/90-routing-sysctl.conf

Install PIP

Securely download get-pip.py.

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

Check whether everything seems fine with get-pip.py. If yes, run the following command:

sudo python get-pip.py

Build FRRouting Packages on Router Server

To install the required packages, run the following command:

sudo yum install git autoconf automake libtool make gawk \

readline-devel texinfo net-snmp-devel groff pkgconfig \
json-c-devel pam-devel bison flex pytest c-ares-devel \
perl-XML-LibXML python-devel systemd-devel

sudo pip install sphinx

Add FRR User and Group

sudo groupadd -g 92 frr
sudo groupadd -r -g 85 frrvt
sudo useradd -u 92 -g 92 -M -r -G frrvt -s /sbin/nologin \

-c "FRR FRRouting suite" -d /var/run/frr frr

Download FRR Source, Configure and Compile

This document assumes that you want to compile and install FRR from Git, and not use any packages.

git clone https://github.com/frrouting/frr.git frr
cd frr
./bootstrap.sh
./configure \

--bindir=/usr/bin \
--sbindir=/usr/lib/frr \
--sysconfdir=/etc/frr \
--libdir=/usr/lib/frr \
--libexecdir=/usr/lib/frr \
--localstatedir=/var/run/frr \
--with-moduledir=/usr/lib/frr/modules \
--enable-pimd \
--enable-snmp=agentx \
--enable-multipath=64 \
--enable-ospfclient=yes \
--enable-ospfapi=yes \
--enable-user=frr \
--enable-group=frr \
--enable-vty-group=frrvt \
--enable-rtadv \
--enable-systemd \
--disable-exampledir \
--enable-watchfrr \
--disable-ldpd \
--enable-fpm \
--enable-nhrpd \
--enable-eigrpd \
--enable-babeld \
--with-pkg-git-version \
--with-pkg-extra-version=-MyOwnFRRVersion

make
make check
make install

Create an Empty FRR Configuration File

sudo mkdir /var/log/frr
sudo mkdir /etc/frr
sudo touch /etc/frr/zebra.conf
sudo touch /etc/frr/bgpd.conf
sudo touch /etc/frr/ospfd.conf
sudo touch /etc/frr/ospf6d.conf
sudo touch /etc/frr/isisd.conf
sudo touch /etc/frr/ripd.conf
sudo touch /etc/frr/ripngd.conf
sudo touch /etc/frr/pimd.conf
sudo touch /etc/frr/nhrpd.conf
sudo touch /etc/frr/eigrpd.conf
sudo touch /etc/frr/babeld.conf
sudo chown -R frr:frr /etc/frr/
sudo touch /etc/frr/vtysh.conf
sudo chown frr:frrvt /etc/frr/vtysh.conf
sudo chmod 640 /etc/frr/*.conf

Install Daemon Configuration File

sudo install -p -m 644 redhat/daemons /etc/frr/
sudo chown frr:frr /etc/frr/daemons

EDIT /etc/frr/daemons

To enable daemons, change noto yesfor watchfrr_enable, zebra, bgpd

Install the FRR Service

sudo install -p -m 644 redhat/frr.service /usr/lib/systemd/system/frr.service
sudo install -p -m 755 redhat/frr.init /usr/lib/frr/frr

Register System Files

sudo systemctl preset frr.service

Start or Reboot FRR Manually

sudo systemctl start frr

Edit the /etc/frr/vtysh.conf File

Add the following line to the file: no service integrated-vtysh-config

Edit the /etc/frr/zebra.conf File

hostname Zebra
password USER_PASS
enable password ENABLE_PASS
log file zebra.log
!
!
!
vrf Default-IP-Routing-Table
ip route 0.0.0.0/0 AWS_PEER_IP
ip route BACKEND_BARE_METAL NETWORK enp1s0f0
interface enp1s0f0
ip address YOUR_PEER_IP/30
!
!
!
!
line vty
!

Edit the /etc/frr/bgpd.conf File

hostname Bgpd
password USER_PASS
enable password ENABLE_PASS
log file bgpd.log
!
!
router bgp USER_ASSIGNED_ASN_AWS_SIDE
neighbor AWS_PEER_IP remote-as AWS_SIDE_ASN
neighbor AWS_PEER_IP password AWS_BGP_AUTH_KEY
neighbor AWS_PEER_IP timers 10 30
neighbor AWS_PEER_IP default-originate
neighbor AWS_PEER_IP soft-reconfiguration inbound
!
address-family ipv4 unicast
network BACKEND_BARE_METAL_NETWORK

exit-address-family
!
!
line vty
!

Reboot FRR Manually

To reboot, run the following:

sudo systemctl start frr

Install Telnet

Telnet allows management of Zebra and BGPd via VTY. To install Telnet, run the following:

sudo yum install telnet

Bare Metal Backend Network

On other machines in your Bare Metal Backend Network, you will need a route to the Router Server.

Edit GATEWAY=“”in /etc/sysconfig/network-scripts for your appropriate backend interface for each server that needs to connect to AWS. Set GATEWAY=“ROUTER_SERVER_IP” where ROUTER_SERVER_IP is the IP address of your Router Server.

Restart networking after making the changes.

Verify the Newly Created Virtual Interface

After successfully establishing a virtual interface with your AWS resources, it is advised to verify your connection using the following procedures.

Verify the Virtual Interface Connection to the AWS Cloud Service

Run tracerouteto verify that the AWS Direct Connect identifier is in the network trace.

Use a Pingable AMI to Verify Your Virtual Interface Connection to Amazon VPC

A pingable Linux AMI, such as Amazon Linux Ami, is a great tool to check your connection to Amazon VPC. Launch your instance of EC2 into the VPC attached to your VPG (i.e., virtual private gateway).

You should see the Amazon Linux AMIs on the Quick Starttab. Make sure that the security group tied to the instance permits inbound ICMP traffic. Once the instance of EC2 is running, get its private IPv4 address (see instance details). Ping that private IPv4 address and check for a response.

Connection Redundancy

Each Direct Connect connection is a single dedicated network between your equipment and an Amazon router. If you need a redundant connection, it is strongly advised to establish a second connection.

Important Notes

IPTABLES on servers in your environment are still in effect and may disrupt traffic flows if not managed correctly.

Security Groups in AWS are still in effect and may disrupt traffic flows if not managed correctly.