Secure SIP Trunks and Secure Conferences with a CA Certificate
Giving yet another nod to the ever-expanding world of security on Cisco equipment, we’re now looking at how to set up secure SIP trunks and secure hardware conferencing using a CA-signed certificate.
As always, the operations detailed in this blog may not be the final word on configuring the items. However, we discovered that there are few, if any, articles dealing with these subjects. We felt it would be a service to offer information about what worked for us in our specific set of circumstances. Comments are always welcome and we will be happy to update these blogs if need be.
In other articles, we’ve described ways to perform all of these tasks using self-generated certificates. Those work just fine, and if the site has a handful of gateways it’s probably best to stick to the steps described in those articles because it’s a lot less expensive; depending on where you get them, certificates can cost quite a lot of money. However, if a site requires dozens of gateways, self-generated certificates cause the following to happen:
As the number of certificates in the CUCM store grows, it takes a very long time to display the certificates when you try to look at them. The reason is that CUCM needs to sort through all the certificates when it is called upon to do almost anything, including the simple act of displaying the contents of the store.
When CUCM has to sort through several certificates, it can also slow down processes that require secured connections. This can mean that LDAP synchronizations can start to fail, and – even worse – it can mean that SIP trunks start to flap.
A flapping trunk can cause the Call Manager service on some versions of CUCM to crash.
If the Call Manager service on these versions of CUCM crash, they can take down trunks leading to other clusters (like an SME cluster), which will then cause those Call Manager services to crash.
If the SME services crash, it brings down the entire system.
Unfortunately, we know this all too well.
Keeping the certificate store in CUCM lean, therefore, is something that will keep engineers from having to answer a lot of late-night phone calls.
Before getting into the guts of this, let’s take a second to understand a few things:
When obtaining a certificate from a Certificate Authority (CA), a certificate signing request (CSR) must first be generated. That request must be submitted to the CA. Typically, more money than one would expect changes hands, and after some time, the CA returns a certificate. It’s usually the case, though, that more than one certificate is sent. There is almost always a separate root certificate plus one or more intermediate certificates as well as the certificate that’s specific to the device. These form a certificate “chain.” The root is thought of as being at the top; the intermediate certificate trusts the root, if there’s a second intermediate, it trusts the first one, and finally the device-specific certificate trusts the last intermediate.When working with servers and routers, both the cluster and the gateways must generate these requests, and both must receive and install the certificates.Note that some CAs want to try to return these certificates in a bundle with a “.p7b” extension. This is workable to some extent, but a pain. It’s easier if the root, the intermediates, and the server or gateway certificates are delivered as separate files.Once the certificates are installed, the cluster can trust the gateway, the gateway can trust the cluster, and everyone can start encrypting in a big, pink bubble of happiness.
It’s a good idea to consider the name of the voice gateway. When dealing with certificates, making sure that names match is critical. A technician at TAC even stated that the case of the letters in the name makes a difference. So, if the voice gateway’s hostname is “verybiggw”, the RSA keys should be named “verybiggw”, and everything else that has anything to do with cryptography from the router’s point of view should be called “verybiggw” too. The exception is the trustpoint, which can be anything, but it’s a lot easier to make the name something like “verybiggw” to keep it consistent throughout.
Find out if it will be possible to have gateways auto-register with the CA. In my experience this is something that can only be done when the CA is on-site as opposed to it being a CA kept at a big corporation. If it’s possible to do auto-registration, it’s worthwhile because the gateways will automatically download the required certificates in one maneuver, and they will be able to automatically update the certificates a few years down the road when they are about to expire. If auto-registration isn’t available, manual installation of the certificates is still possible and worthwhile. We’ll detail both methods here.
People who are real “smarty-pants-types” might consider creating their own CA. It can be done on practically any Windows server. Not only does this make auto-registration possible, it also saves a lot of money on certificates, and the certificates are available on-demand. The reason why this shouldn’t be a hard-sell in most organizations is because the certificates will only be used to encrypt communication between CUCM and the gateways, nothing is user-facing.
It doesn’t matter if your gateway will be configured with manually downloaded or auto-downloaded certificates, the first step in the process is to get the certificate signing request from CUCM. Go to the OS Administration page on the Publisher and navigate to Security > Certificate management. Click the button that says, “Generate CSR”. In the new window’s Certificate Purpose drop-down select Call Manager.
This is where it gets a little tricky. It’s worth cultivating a working relationship with your friendly CA person because they are the ones who will be able to help you with the formatting of the request.First find out if it will be possible to get a Multi-server (or Multi-SAN) certificate. If so, do it. This will cause the CA to generate a single server certificate that can be installed on the Publisher. The certificate will then be pushed out to all the rest of the servers in the cluster. Without this, a CSR will have to be generated on every server in the cluster. Not only is that more work, it’s also more expensive. These days most CAs allow for Multi-SAN certificates, so that’s what will be covered here.
Go to the distribution drop-down and select Multi-server (SAN). Then wait a second for the rest of the fields to populate.Next, find out what common name the CA wants. By default, Cisco uses “<FQDN of Publisher>-ms”. It’s my experience that CAs want the FQDN only, so we usually have to erase the “-ms” part. We did, however, have a CA that wanted us to put a code number in that line, so make sure this question is asked of the CA people.
Next, discover what the CA wants in the Parent Domain field. The CAs we’ve used usually want that left blank.Finally, find out from the CA what the Key Length and the Hash Algorithm should be. By default, CUCM populates those fields with “2048” and “SHA256” respectively. If the CA wants anything less than those, get a new CA.
When the fields are populated, click Generate. When the CSR is generated there will be a prompt saying as much. At that point click close.
Click Download CSR and save it somewhere logical.
Manual Installation of Certificates on Gateways:
First, configure secure calls (as detailed in the article “Configuring Encryption on Cisco IP Phones”). If your phones can’t handle encryption, you could still make a case for securing trunks, but not conference bridges.
On the gateway, generate an RSA key. Type:verybiggw(config)# crypto key generate rsa modulus 2048 exportable label verybiggw
Next, create a trustpoint for the certificate, configure enrollment for it, and associate the RSA key to the trustpoint. Type:crypto pki trustpoint verybiggw enrollment terminal rsakeypair verybiggw
The next configurations are supposedly optional, but recommended. We should mention, however, that we have yet to see a CA that didn’t want them in there. These configurations set the domain name of the certificate. Check to see if the certificate has been revoked, put the name of the router in the subject of the certificate, and say that no serial number or IP address should be in the certificate. Type (still under the trustpoint):fqdn verybiggw.verybigco.comrevocation-check none subject-name CN=verybiggw serial none ip-address none
The next configuration enrolls the certificate. When this command is used, it creates the gateway’s certificate signing request that will be submitted to the CA. Copy the information that is displayed, but (unlike actual certificates) only copy the block of numbers and letters that looks like coded gobbledygook. Do not copy the line that says “—-END – This line not part of the certificate request.” Type:verybiggw(config)# crypto pki enroll verybiggw
When we did this, we got a certificate request that looked something like:MIIBNTCB4AIBADBaMSgwJgYDVx9NVEIwMTJDUkUzOTI1UEJYMS50aWVyNC5p
Copy the information from the screen, paste it into Notepad (really use Notepad here so it doesn’t do any formatting) and save it with a “.csr” extension.
Take the CSR from the servers and the CSR from the gateway, submit it to the CA, write a check for what is honestly a criminally large sum of money, and wait for the certificates to arrive.
While waiting for the certificates, it will be possible to configure the gateway. Configuring the conference bridge is basically the same as configuring a normal conference bridge, but there are a couple of pitfalls. Start with the voice-card. Type:voice-card 0dsp services dspfarm
Next, do the profile. Notice that the bridge must be associated with the security trustpoint, and that a “no shut” is required:dspfarm profile 1 conference securitytrustpoint verybiggw maximum sessions 4 associate application SCCP no shut
Configure SCCP, but add the trustpoint:sccp local gigabitEthernet 0/0sccp ccm 10.1.1.1 identifier 1 version 7.0 trustpoint verybiggw sccp
Configure the SCCP Group and register the name of the conference bridge. This is something that trips up a lot of people. Normally, the name that is used for registration can be just about anything. However, since certificates are being used, and certificates care a lot about nomenclature, the name that’s used here must match the router name.sccp ccm group 1bind interface gigabitEthernet0/0 associate ccm 1 priority 1 associate profile 1 register verybiggw
For the SIP trunk, there are a couple of things that are a bit different than the configuration of a typical SIP trunk. One is that the SIP-UA must be associated with the security trustpoint. Type:sip-uacrypto signaling default trustpoint verybiggw
The other thing that is different is that dial-peers require a couple extra configurations. Both the inbound dial-peers coming from CUCM and the outbound dial-peers going to CUCM require the following to secure the signaling and secure the actual voice packets:dial-peer voice 100 voipsession transport tcp tls voice-class sip srtp negotiate cisco
At this point, the basic voice gateway configuration is complete. Assuming the certificates haven’t arrived yet (because they won’t have) it will be possible to configure CUCM. We’ll start with the conference bridge. This configuration is the same as most IOS Enhanced Conference Bridges, the only differences are that the name must be the hostname of the gateway (because it’s the name of the bridge), and in the Device Security Mode drop-down, the selection must be Encrypted Conference Bridge.
Before configuring a SIP trunk, a SIP Trunk Security Profile must be created. Navigate to System > Security > SIP Trunk Security Profile and create a new one. Give it a name and a description. In the Device Security Mode drop-down, select Encrypted; this should auto-populate the Incoming and Outgoing Transport Type fields with TLS. The X.509 Subject Name should be the FQDN of the gateway. The Incoming Port should be 5061. Finally, there are a bunch of check boxes that should be selected based on the environment. What we’ve used with success is a checkmark in Accept presence subscription, Accept out-of-dialog refer, Accept unsolicited notification, Accept replaces header, and Transmit security status. We’ve always checked all, but the last one in the list because they’re required for Presence; they won’t affect security, but it won’t hurt anything to have them checked. The last check box, though, definitely is required for encryption.
On the actual SIP trunk, there’s a box next to SRTP Allowed that should have a check in it, the “Consider Traffic on This Trunk Secure” drop-down should read “When using both sRTP and TLS.” In the Destination area, the Destination Address should be the IP of the gateway (as normal). To the right of the Destination Address, the Destination Port should be 5061. Finally, the SIP Trunk Security Profile should be the one configured in the previous step. Save it all and reset the trunk.
When the certificates (finally) arrive, they will probably have to be modified. Remember that if the certificates all come in a single “.p7b” bundle, you’ll have a difficult time making them work. We’ve managed to get them to work if the bundle only included the root and intermediate(s), but never if the bundle also included the device certificate itself.If the root, intermediate, and device certificates come with the “.cer” extension, change them to a “.pem” extension. This can be done by simply renaming them (this won’t mess up anything). Note that sometimes a certificate with a “.cer” extension can be uploaded to CUCM, but we’ve also seen them fail. Make it easier on yourself and change the extension.
Next, upload the certificates to CUCM. Go to the OS Administration page, navigate to Security > Certificate Management and click Upload Certificate/Certificate Chain. Upload the root first. The Certificate Purpose field should say “CallManager-trust.” The Description can be left blank. Then, in Upload File, browse to the root and upload it. Do the same for each intermediate certificate. When this is done, replication should shoot the certificates to all the servers in the cluster, so there’s no need to repeat this step on every server.
Finally, upload the device certificate that came in response to the CUCM CSR as a Call Manager (as opposed to CallManager-trust) certificate. Since a Multi-server certificate was ordered, this will cover all the servers, so there’s no need to install a server-specific certificate into each server.
The appropriate certificates must also be installed on the gateway. Get the certificates that came in response to the gateway’s CSR and open them in Notepad. In this case, the contents of each certificate will need to be copied and pasted into the gateway. When doing the copy, make sure to include everything on the certificate including —–BEGIN CERTIFICATE— through all the numbers and letters and all the way to the end of —–END CERTIFICATE—–. Notice that the BEGIN and END statements and all the dashes must be in there.
On the gateway, enter:crypto pki authenticate verybiggw
When this is done, a prompt will be issued saying that the certificate must be pasted into the gateway after which there should be a space, then the word “quit.” What we’ve done with gateways is we’ve just copied the last intermediate certificate only and put it in.What do we mean by, “the last one?”
Remember the certificate chain mentioned above. If there’s more than one intermediate certificate, the one that must be placed in the gateway is the one trusted by the gateway-specific certificate. To figure this out, it’s possible to go to that gateway certificate, change the extension to “.cer”, then double-click it to open it. A new window will pop up. Click on the Certificate Path tab. The final intermediate certificate shown in the chain is the one that should be copied into the gateway.
Don’t forget to change the extension back to “.pem”.
Select the correct intermediate certificate, open it in Notepad, copy the contents and paste it all into the gateway. Hit Enter, type “quit”, and hit Enter again. A prompt will come up asking if the certificate should be accepted. Type “yes”.
Still on the gateway, enter:crypto pki import verybiggw cert
This will show the prompt asking for the certificate for the gateway itself. Make sure the certificate has a “.pem” extension, open it in Notepad, then copy it and paste it into the gateway. Hit Enter, type “quit”, and at the final prompt type “yes” to accept the certificate.
That should do it. To test secure conferencing, set up a conference using secure phones. Once in a conference, the padlock should be displayed next to the caller ID. Once the conference is set up, it’s also possible to go to the router and issue the command “show dspfarm dsp active”. The TYPE column should show something like “s-conf 1” where the “s” stands for a secure conference.
To see if the SIP trunk is secure, place a call that travels down the trunk to the gateway. The padlock should also be seen next to the caller ID. If you didn’t put encryption onto the phones, you can test the connection by making sure there’s an OPTIONS ping configured in the trunk’s SIP profile in CUCM, then going into the gateway and typing:show sip connection tcp tls brief
If the trunk is working, the number of connections should be something greater than zero.
What about the fancy Auto-Registration of Gateways?
Remember that auto-registration simplifies things for administrators by eliminating the need to create a CSR on the gateway, and eliminating the need to manually revisit every gateway after a few years to update the certificates. All of this is achieved by changing the trustpoint on the gateway.
What you’ll need from your CA is the URL of the CA server to which you’ll connect, and some sort of password that will be required to start the download.
Notice in my example below, we’ve also configured an “auto-enroll” line, which tells the router to re-download a new certificate after “X” percent of the lifetime of the current certificate has elapsed. In my example, it’s 80%.
Replace the trustpoint in the previous configuration steps with something that will be very close to the following:crypto pki trustpoint verybiggw enrollment url http://<url_provided_by_ca> serial-number none fqdn verybiggw.verybigco.com ip-address none subject-name CN= verybiggw.verybigco.com revocation-check none source interface GigabitEthernet0/0 rsakeypair verybiggw password <the_password_provided_by_the_CA> auto-enroll 80
Next, enter:crypto pki authenticate verybiggw
And then:crypto pki enroll verybiggw
The certificates should download and install themselves. If you issue “term mon” at the privileged prompt before you type those last two configurations, three or four alerts will be generated that will tell you if the download and the installation was successful. Then if you issue “show crypto pki certificate verbose” you’ll be able to see the details of the certificates themselves.
Comments are welcome, both in agreement or constructive disagreement about the above. I enjoy hearing from readers and carrying on deeper discussion via comments. Thanks in advance!
A Cisco Certified Networking Professional (CCNP) for voice, as well as routing and switching. Paul has specialized in Cisco networking solutions since 1996 and has been focused on Cisco voice products since 2007. He is also a Cisco Certified Systems Instructor (CCSI #31022) and continues to develop knowledge transfer sessions for voice as well as routing and switching.
Nick has over 20 years of experience in Security Operations and Security Sales. He is an avid student of cybersecurity and regularly engages with the Infosec community at events like BSides, RVASec, Derbycon and more. The son of an FBI forensics director, Nick holds a B.S. in Criminal Justice and is one of Cisco’s Fire Jumper Elite members. When he’s not working, he writes cyberpunk and punches aliens on his Playstation.
Virgilio “BONG” dela Cruz Jr.
CCDP, CCNA V, CCNP, Cisco IPS Express Security for AM/EE
Field Solutions Architect, Tech Data
Virgilio “Bong” has sixteen years of professional experience in IT industry from academe, technical and customer support, pre-sales, post sales, project management, training and enablement. He has worked in Cisco Technical Assistance Center (TAC) as a member of the WAN and LAN Switching team. Bong now works for Tech Data as the Field Solutions Architect with a focus on Cisco Security and holds a few Cisco certifications including Fire Jumper Elite.
CCIE #1066, CCDE #20070002, CCAr
Chief Technology Officer, Practice Lead Security Services, NetCraftsmen
John is our CTO and the practice lead for a talented team of consultants focused on designing and delivering scalable and secure infrastructure solutions to customers across multiple industry verticals and technologies. Previously he has held several positions including Executive Director/Chief Architect for Global Network Services at JPMorgan Chase. In that capacity, he led a team managing network architecture and services. Prior to his role at JPMorgan Chase, John was a Distinguished Engineer at Cisco working across a number of verticals including Higher Education, Finance, Retail, Government, and Health Care.
He is an expert in working with groups to identify business needs, and align technology strategies to enable business strategies, building in agility and scalability to allow for future changes. John is experienced in the architecture and design of highly available, secure, network infrastructure and data centers, and has worked on projects worldwide. He has worked in both the business and regulatory environments for the design and deployment of complex IT infrastructures.