Register for our next C-MUG Meeting "The Four Kinds of Clouds, and How Each Affects Your Business" on August 19th!

4/6
2010
William Bell

Audit Logs on Cisco Unified Communications Manager (CUCM)

Background

I will keep the background discussion to a minimum, because I am sure most readers of this blog understand the need for an auditing solution.  To summarize, admins need a way to look at an audit trail for administrative changes that occur in their CUCM cluster.  Basically, in large organizations there is a need to look at “what has changed” so that if there is an issue reported that is related to human error, the error can be quickly identified and fixed.  There is also an accountability aspect as it is likely that senior support folks want to identify education gaps in their lower tier support teams.

Software Revisions

The ability to enable auditing for administrative tasks was included in CUCM version 7.1(2) and later.  In my lab, I tested this feature using a CUCM 7.1(3b)SU2 cluster.

Enabling Audit Logs

By default, audit logs for administrative level tasks are enabled.  To access this feature, you will need to browse the audit log parameters from Cisco Unified Serviceability.  The following procedures can be used to accomplish this task:

1. Connect to Cisco Unified Serviceability on your CUCM cluster: https://mypublisher/ccmservice/

2. Go to Tools>Audit Log Configuration

3. Depending on your particular operational model, you may have people with different levels of authorization.  Authorization levels are implemented via the Groups and Roles configuration objects.  If you have a need to customize roles or Group/Role assignments you will want to look at the “Standard Audit Log Administration” role to understand the level of access required for users/groups that need to look at audit logs.

Accessing Audit Logs

At this point, you have enabled audit logs for administrative tasks.  As with other logs and trace files on the system, you can access these logs in several ways.  For this example, we will walk through using RTMT to view audit logs.

1. Load RTMT on your admin workstation

2. Connect to your CUCM cluster using a user ID that has the “Standard Audit Log Administration” role (or equivalent) assigned.

3. Go to System>Tools>Trace>Trace and Logs Central

4. Under “Trace and Logs Central”, you will see an “Audit Logs” folder.  Double click the folder to start the log collection wizard.

One of the benefits in using RTMT is that you can collect and download AuditLogs either as a one time request or you can schedule the download.  For those that like to run pro-active analysis (via scripts and the like) you may want to look at the “Schedule Download” option.  As with other traces/logs, once you specify the search criteria RTMT will provide a folder tree structure that can be used to navigate to the audit logs stored on the server.

 

Deciphering the Audit Log

The log file entries are lengthy but parsing them is not at all difficult.  Let’s look at an example.  I have a dummy phone on my CUCM cluster.  The phone was already created prior to testing the audit log feature.  What I did was add a phone number to the device.  In the AuditLog this action is captured as follows:

04/06/2010 11:03:15.295 |LogMessage
UserID :ccmadministrator
ClientAddress :192.168.1.4
Severity :5
EventType :GeneralConfigurationUpdate
ResourceAccessed:CUCMAdmin

EventStatus :Success
AuditDetails : record in table numplan with key field dnorpattern = 2025552222 added
ComponentID :Cisco CUCM Administration
App ID:Cisco Tomcat
Cluster ID: Node ID:iecucm01

Now, the log entry itself will be a single line in the trace file but I have broken it out in something a tad more legible.  You will see that the log identifies the user who performed the change (UserID), the IP address of the machine the user was using (ClientAddress) along with other pertinent information.  Of particular interest is the AuditDetails section of the logged event.  In this example, we can see that I added a directory number.   Before dissecting this section, we should address an obvious question: which device was this performed on?

This is where things are tricky.  If you look at previous log entries, you will find when the user (ccmadministrator in our example) connected to the CCMadmin web portal.  However, the AuditLog will not track this user’s movements through the various web pages.  So, if I clicked on a device, and then clicked on “Add new line”.  Those events are not recorded.  Only when I actually “save” the change will it be recorded in the AuditLog.

Why?  Well, it boils down to the fact that the Audit Log is tracking changes to the database.  It’s focus is on what has changed in the database, not what has changed in the admin interface.  This is actually a necessary distinction, since changes to the database can come from a variety of sources:  BAT, CCMAdmin, AXL/SOAP (e.g. UCCX creates a JTAPI trigger).  If you look at our example above, you will see that the AuditDetails says:

record in table numplan with key field dnorpattern=2025552222 added

Numplan is a database table and if we wanted to find out what device this number was added to, we would need to take a few extra steps.  One step could be to run an ad hoc query from the command line to see what gives:

admin:run sql select d.name as device,n.dnorpattern,dmap.numplanindex
from numplan as n inner join devicenumplanmap as dmap on dmap.fknumplan=n.pkid
inner join device as d on dmap.fkdevice=d.pkid
where n.dnorpattern='2025552222'
device dnorpattern numplanindex
=============== =========== ============
SEP00258418EE10 2025552222 3

From this output, we have identified the device and the line appearance the new number is assigned.  Yeah, I know the query is ugly.  I can’t resist you see, I do most of my admin tasks via custom queries.  That’s just me.  You could certainly figure this out via the CCMAdmin web pages by going to Call Routing>Route Plan Report.  Then search for the directory number “2025552222”.

So, you still have some work to do when you want to analyze the audit log, but the data now available is a great improvement over what you would have to do in the past.  I won’t even waste your cycles on that discussion.

Other Things That Are Logged

As you play around with the Audit Log, you will find that it captures other interesting things as well.  Such as commands entered by users on the CLI.  This is handy if you want to find out who restarted the TomCat service or who reloaded the server.  Small things like that.  Of course, if you want that data to have any meaning, you may want to investigate the CLI command: set account name.

The Audit Log will also capture events like a user uploading a custom banner page to the CUCM system.  The custom banner page is another new security feature added to CUCM 7.1(2).

Where is the Audit Log

For those folks who don’t dig on using RTMT for certain tasks, and would just rather download the file or view it directly on the console I have two pieces of information:

1. The files are stored here:  activelog audit/AuditApp/*

2. If you go this route, I recommend downloading the files to your workstations and then viewing/parsing/filtering.  Especially during production hours.

Conclusion

Well, the AuditLog feature isn’t 100% perfect as you still have to tear yourself away from CNN or ESPN to actually do some work.  But, it is getting there and the information that is recorded in the AuditLog is very handy when you need to determine the “how”, “when”, and “why”.  Hmmm, maybe it won’t tell you the “why” but it sure will point you to the person that can!

William Bell

William Bell

Architect, UC&C Practice Lead

William’s background spans an array of technical disciplines including application development, network infrastructure, protocol analysis, virtualization, and Unified Communications. Bill is certified as a CCIE Voice (CCIE #38914) and possesses a deep understanding of Cisco’s UC and Collaboration portfolio. He leads the UC engineering team and also works with customers on architecting solutions that align with core business drivers. Bill is a regular contributor on the Cisco Support Community, a 3 time Cisco Designated VIP, and blogs on the NetCraftsmen and UC Guerrilla sites.

  • Steve

    Appreciate your detailed and insightful tutorial. It’s very timely because I’ve been searching on and off for the last few weeks for a method of auditing changes to a 6.0(2) cluster. I was wondering if there may be a way of tracking or capturing tomcat login and page serves for a specific user account. Can you think of a trace or capture that might accomplish this?
    Humbly,
    Steve Sutton

  • William Bell

    Steve,

    Sorry for the delay in getting back to you. I missed this comment in my e-mail stream. The audit logs outlines in the article can be used for this purpose. Back in the day, I would use the IIS logs (pre-appliance model) to do something similar but that was a chore. You can see similar data with tomcat. On the CUCM system (usually the publisher node) you can view a daily access record for the tomcat service:

    [b]admin:file list activelog tomcat/logs/localhost* detail[/b]

    You can then view one of these files to see if it is close to what you need. I am not sure if there is any more value in tomcat that the audio logs don’t already provide but you can determine what best meets your requirements. (NOTE: You can also browse the tomcat logs using the same methods described in this article).

    Now, if you are looking for a way to pull or push the traces off box and then post process using perl or something similar, then you have a few hoops to jump through. You can push tomcat traces to a syslog server. Take a look at the CUCM Serviceability configurations (Alarm configurations). Tomcat traces can be configured under the platform services for your server.

    I am not sure if there is a way to push auditlogs to a syslog server but you can pull them on a scheduled basis using RTMT. There are other methods, but using RTMT is pretty straight forward.

    Regards,
    Bill

  • Guy Kelly

    Hi William,

    Actually this article is incorrect. The logs you are enabling above are the database logs and are filed under the informixauditlogs on RTMT. Application audit logs are enabled by default and are the checkboxes listed in the Application Audit Log Settings above.

    Cheers
    Guy Kelly

  • William Bell

    Guy,

    Yup, you are absolutely correct. I have nothing to say except thanks. I have made the correction and plan to come back and expand upon blog later.

    Thanks,
    Bill

  • Vladimir

    Hi!
    Thank you for this very usefull article!
    Is there any way to delegate to any user a permission to manage only some pool of phones?
    In large organisation with many Regions anf offices we have to give to local admins roles:
    Standard CCM end users
    Standard CCM phone management
    Standard CCM user management
    Every thing works fine, local admin can add and change phones in his office.
    The problem that ye can can change and delete ANY phone in Organization.

    Is there any way to resolve it?
    Thanks
    Vladimir

  • William Bell

    Vladimir,

    The behavior you are observing is by design. The CUCM authorization model only provides for what web pages you can access and the permission level on those pages. There is no way you can provide for a mult-tenant admin model using the native CUCM software.

    There are third-party applications that address this need. I haven’t looked at Cisco’s Unified Provisioning Manager, but it may provide what you are looking for (and then some). You could also look at Unimax, 2Ring, and similar partners. I cannot recommend a solution on this site. Perhaps query: "cucm multi-tenant provisioning" or "cucm multi-tenant administration" in Google.

    HTH.

    Regards,
    Bill

  • ahmed

    commenting on response of :William Bell, February 24, 2011 ,

    Really this is good info ,,,pls include some examples sql commands ,
    i appreiciate for good work :)
    :)

  • William Bell

    Ahmed,

    I have a series that goes through various SQL queries. This series is hosted on my personal blog: http://ucguerrilla.com.

    HTH.

    -Bill (@ucguerrilla)

  • Sally

    I find that I still want to shoot myself while looking at log files.

  • Thomas

    Nice Post.. Excellent Info.. Really amazing.. This was a fantastic article… really superb….

View more Posts