ADT Security is installing a CEM Systems’ Linux-based AC2000 AE IP access control solution managed by AC2000 management software in a virtual environment at Auckland Airport in New Zealand. The system comprises 1000 doors and more than 14,000 cardholders.
THERE’S no way to overstate the multi-layered complexity of airport electronic security applications. These are rarely Greenfield sites. Upgrades are shaped and constrained by existing infrastructure and by the nature of systems that came before them. Airports are built concentrically over years, like the rings of a tree. Under their glittering glass and steel skins lie core services cemented into place decades before.
There’s a necessary political side to major airport security solutions. September 11, 2001 changed airport security forever and showed us unequivocally that a serious security breach at one airport paralyses global flight. Given its vital national and international importance, multiple internal and external government, public and private organisations are tasked with governance and support of security operations at Auckland Airport. This complicates the process of security system design and implementation.
Overarching all these burdens is the fundamental challenge imposed by the physical nature of an airport. Across the 1500Ha site are hotels, car parks for 6500 vehicles, workshops, fuel storage facilities, administration buildings, huge airport terminals and all the rest. Remote cable runs here are measured in kilometres, significant buildings are numbered by the dozen, there are 1000 access readers and 14,000 cardholders.
Auckland Airport operates 24 hours, 365 days a year. It’s the second busiest airport in Australasia – only Sydney is busier – and it handles more than 14 million passengers annually (nearly 40,000 passengers each day). During the upgrade process there can be no loss of capacity, no breakdown of the access control procedures managing its 14,000 employees and contractors.
That means doors must be cutover one at a time over a long period, or the new system must be installed and commissioned in parallel with the old, then the entire site cutover as quickly as possible. These challenges and more are being faced by Auckland Airport project manager Andrew Catterall and the ADT integration team supporting his drive to haul this huge site’s access control infrastructure firmly into the digital age.
Planning the upgrade
A howling 25-knot Westerly sweeps spatters of rain across Manukau Harbour as I arrive at Auckland Airport with Tyco’s Scott Whitehead. As we negotiate the big car park, I glance at the temperature gauge on the dash – it’s 4C degrees outside. Burrowing through the gale to Auckland Airport’s administration building, Whitehead, who has been in the thick of this application for 3 years, shouts the challenges of the site at me over his shoulder.
According to Whitehead, when Auckland Airport decided to replace its 15-year-old Honeywell access control solution, the plan was to retain the existing serial infrastructure. This may sound unusual but on a site of such magnitude there’s a sound financial case for holding onto existing copper. But as the contract began to unfold it became clear there were going to be issues with this approach. The entire project was put on hold while Airport chewed over its options and came to a decision. It would go full IP.
“The project has been boiling along for 2 years – domestic was the first to switch from serial to IP because of the relevant upgrades being carried out in the domestic terminal,” Whitehead explains. “Then the airport made the decision near the end of the domestic roll-out to change the international terminal to IP.”
Andrew Catterall, project manager, Auckland Airport
Inside the sudden calm of the admin building, I sit down with Andrew Catterall, project manager, Auckland Airport. Catterall is an English expat who’s lived in New Zealand for 20 years. Union Jack cufflinks suggest he’s not forgotten the old country but while he references bustling Heathrow as an exemplar of airport security excellence, Catterall well and truly owns Auckland Airport’s evolving security solution.
“We did a lot of research on what access control solution we wanted,” Catterall tells me. “One of the main things was that the new system had to be future-proof and highly utilised – we did not want to be guinea pigs. We looked at a number of systems but CEM was the one that came to the top – it’s used in 120 airports worldwide and airside clients are comfortable with it.
“The project has been boiling along for 2 years – domestic was the first to switch from serial to IP because of the relevant upgrades being carried out in the domestic terminal”
“We approached the decision asking ‘what do the high risk airports use and why?’ We found CEM was deployed across almost all UK and many European airports, as well as key international air hubs like Heathrow, Hong Kong, Dubai,” Catterall says. “During the process I asked manufacturers ‘what can you do for an airport?’ We decided that the CEM Systems’ solution could do the most for us.
“We liked the fact CEM readers stored the database offline and could carry on working at the local doors. With a traditional access control solution, if the serial cable is cut between system controller and door controllers and readers, then the system is dead.
“And we appreciated that software support was handled in Belfast,” he explains. “Most software problems are dealt with after working hours and the internet has made the world smaller. It’s better for a company to be dealing with you in their daytime when they have the full complement of programmers, rather than bringing in tired people after hours who aren’t thinking straight and may not be supported properly.”
The IP decision
At this point in the project, Auckland Airport was still planning to install CEM readers onto the site’s existing serial cable. But at a key juncture the entire project team had a moment of mutual realisation. The decision was made to put the project on hold and re-consider options. Next came a visit to CEM Systems in Belfast to talk to the manufacturer about system capabilities and future plans.
Unsurprisingly, Catterall tells me that the catalyst for this decision was something the project team had always been nervous about – the process of transition and how the nature of the existing cable infrastructure would impact upon it.
“The problem with this airport and many others is that the terminals are built around the shells of older buildings – which have been added to and modified,” he says.
“Those old buildings had dictated the layout of the original serial access control LAN and its subsequent organic growth,” Catterall says. “What happens on a serial access control LAN is that you have a number of series drops supporting multiple doors, and technicians tag into the nearest series drop as they work.
“Over time, organic growth across the site led to doors being installed on the same serial drops that were on different floors and in different parts of the building. This mean that during a transition when we took out a controller we would lose doors on random floors and have to manage that challenge across the entire site.”
According to Catterall, the issue led the project team to ask itself whether hanging onto the serial infrastructure was the right approach after all.
“Were we future proofing ourselves against expansion of Auckland Airport by retaining the serial cable?” he asks. “And during a visit to the Belfast factory we learned that the latest CEM Display Readers would only be available on IP”
“That made up our minds,” Catterall says. “We decided we would go IP, we would build a cloud-based system, we would install the new system in parallel with the existing access control solution and we would do all commissioning with the system live but not actually controlling the doors.”
Catterall says the project team applied this model as a test case to the Domestic Terminal Building (DTB), which comprises about 15 per cent of overall door numbers. That test was successful.
“As part of the procedure, we informed all interested parties, we did all the commissioning, everything was tested – all we were doing was moving over the wires that powered the electromagnetic locks which were working with the existing system,” he explains.
“What was beautiful was that when we cutover the DTB system we heard nothing from the airlines. This was a clear sign of success because the airlines were worried about the process. A failed access control system can stop an airport dead if it’s done wrong.
“Having completed the DTB as a test case we could see we had the right approach, that our processes worked, and we could go on and apply the same method to the much bigger International Terminal Building. It also allowed us to see that our initial plan to retain the serial infrastructure, which had seemed right at the time, was actually the wrong approach.”
Old and new readers installed in parallel
Catterall says the team learned a lot from the DTB and is now applying those lessons to the Internal Terminal Building, which is 75 per cent of the site’s access points. And the speed of the DTB cutover streamlined the entire process of upgrade across the site.
“If we had gone serial we would have needed security officers on doors we were taking off-line” he explains. “That would have increased the time needed and clashed with airport operations – so the process would have demanded night work. With this parallel integration we could do the cutover at any time.”
The 500-pound gorilla
Catterall says a key issue with the old system that needed to be resolved before the new CEM solution was installed on the site was the current database, which over many years had gained 4000 different access levels assigned to 14,000 authorised cardholders. These access levels had to be rationalised in communication with all stakeholders.
“The database had just organically grown – people had added new levels to make the old system work – it wasn’t anything anybody did wrong so much as technicians trying to keep the system operating in an organic environment over a period of 15 years,” Catterall explains.
“A big airport like Heathrow with close to 80,000 cardholders has 500 access levels but a typical airport should have 200 levels – that’s where we needed to be,” he says.
The first attempt to rationalise the database carved access levels down from 4000 to 1700 but that was still far too many.
“Our team, led by ADT’s Paddy Browne, did its best to strip the database further but thanks to the time constraints it became a bigger monster than they ever believed it would,” Catterall says. “Making things challenging, too, was that these 1700 access levels no longer had names – they were represented by a number and staff needed an index sheet to decode the numbers.”
The process of stripping back a live database from 1700 to 200 was ingenious, despite seeming counterintuitive. Rather than building another database on another server, re-registering every cardholder then dishing up new cards, the team built a new database inside the old one using a simplified group of new access levels.
“We decided that during the building of the new DTB system, the team would build a mini database attached to the old database,” Catterall explains.
“The process was that 20 new access levels were assigned to the old database and instead of being assigned to one of 500 DTB access levels, new cardholders were grouped into one of the 20 new levels.
“More recently, the new database has been increased by another 180 access levels and we have started transferring existing cardholders from any of the various 1700 access levels in the existing database to one of the 200 access levels in the new database. At the end of the project the old access levels will be simply be carved off.
“The idea was that existing cardholders stayed in the old database but anybody new now went into the new smaller database. There was no struggle to find what level to assign new staff, all the new people just went into the simplified new database,” he says.
According to Catterall, the levels reflect the physical reality of the site. If a cardholder is given access to a door that allows them access to the airside of any other doors, then they are assigned access to all those doors from the inside.
“It sounds simple but in the past doors were assigned in a particular area to cardholders from a particular organisation despite the fact they all accessed the same areas. Once we applied logic to the database the access levels started tumbling in on themselves.”
System and network
To my mind, the network side of the Auckland Airport solution is the key to understanding the nature of the system. This is no typical access control application. Catterall tells me CEM had never installed one of its IP-based systems in a virtual environment and I’ve certainly never seen or heard of one. As Catterall explains the nature of the system I find it hard to conceive the topology without a schematic.
At its heart the CEM access system comprises clever S610e card reader/controllers that have RJ-45 and local power connections. Incorporated into each reader is the database authorised to access the local door, carrying smarts right to the very edge.
Managing remote readers are real time virtual controllers and AC2000 main server (CDC) with integrated backup software Hot standby (failover CDC) and RTC Ethernet controllers, both operating in a virtual environment. The servers talk between each other and incorporate failover and redundancy. Being in a virtual environment means any changes made to the system are automatically replicated to every virtual server.
Meanwhile, each of the real time controllers (RTCs) can support 256 network connected reader/controllers. These RTCs are the firmware-based servers controlling all the alarms and events and IP config. Traditionally, these RTCs would be set up on a hard server but the airport wanted them located in a VM cluster replicated in 4 different areas for disaster recovery.
Because there are no door controllers, power is centralised in hub rooms, with each 16Amp power supply carrying up to 5 local doors. These hub rooms also contain the network switches with assigned ports which carry the readers onto a dedicated VLAN.
CEM Systems’ AC2000 AE solution is an airport-specific access control system and features including check-in desk enabling, passenger mode and air-bridge monitoring. The solution at Auckland Airport uses AC2000 software modules including CEM’s AC2000 VIPPS (Visual Imaging and Pass Production System), which allows security personnel to produce passes and access rights for staff and visitors.
“We decided we would go IP, we would build a cloud-based system, we would install the new system in parallel with the existing access control solution”
The Airport is using AC2000 WEB Visitors which will enable system users to request and manage temporary cardholders (visitors) via a standard web browser. AC2000 T&A (Time and Attendance module) offers easy to read reports of employee In and Out times. The Airport is employing AC2000 Failover to create a system with a high level of redundancy thanks to its ability to create and maintain a copy of the main AC2000 server.
The use of CEM portable card readers is central to the security upgrade at Auckland Airport. These readers enable on-site airport security staff to validate cards at remote sites or areas with no mains power supply and they can also be used as mobile devices for random checks in emergency evacuations. The readers offer the flexibility of setting up controlled access points instantly, without having to physically commission a fixed access controlled door.
As Catterall explains it, the big deal hidden under these specifications is that instead of having 2 physical servers and 6 physical RTC? units, Auckland Airport has 2 virtual servers handling operations while 6 more virtual servers support real time data. All these virtual servers exist in multiple locations in the Airport’s cloud.
“In terms of topology, the system is located onsite in VM clusters in 2 separate locations, one primary server over there,” he points out the window into the distance, “And a secondary server over here in another building. System monitoring is located in an operations facility over there and there’s an emergency operation centre located in yet another building.
“The important thing here is that there’s no dedicated system hardware – the controllers are living in an internal cloud,” Catterall says. “That was a new thing for CEM to approach – how to handle failover in a virtual environment.”
According to Catterall, CEM modified its 6.7 software for Auckland Airport.
“We were the first CEM customer that asked to put our servers in a virtual environment,” he explains. “We also asked CEM to give us escalation of alarms like landside/airside doors to a supervisor’s workstation if they were not actioned by operators in a given time.”
The installation
After chatting with Catterall, Whitehead picks me up in the car and drives me round to the Tyco Project Office in the Domestic Terminal Building to meet engineer Paddy Browne and project manager, Roger Read. Browne and Read have had coal face involvement with the project since the start. As Browne tells his side of the story I get a growing sense of the geological nature of the site, its cable trays, rack rooms, risers and power supplies laid down over years.
“For the installers, the decision to go with an IP solution was a relief,” says Browne. “With IP, each device is an IP address. It doesn’t matter where that device is, you can group it in the most convenient way. That was a huge advantage on a big site like this – much cleaner.”
Browne explains that because CEM Systems’ solutions are designed for airports – the team is not running innumerable scripts to try to make the system work in an airport – in fact the DTB CEM application is virtually default.
“In terms of system layout on the new system, data from reader controllers is ported onto the network to virtual servers and virtual RTCs,” he says.
“There are many advantages to this topology. For instance, when you assign configured access levels to personnel or change an access level or any detail, the change propagates across the entire system automatically.
“That’s excellent from a technician’s point of view. If you fix something on one reader, it’s a system-wide fix. With controller-based systems, if there’s a database problem, you lose all the doors attached to that controller. That can’t happen with this IP-based CEM system.”
“We also asked CEM to give us escalation of alarms like landside/airside doors to a supervisor’s workstation if they were not actioned by operators in a given time”
Once the decision to go full IP was made Auckland Airport’s IT department became much more involved in the installation process. According to Browne, IT provides all the network components and the VM cluster, while ADT designs and install the Linux firmware.
“In the field we simply tell them how many switch ports we need and where,” Browne explains. “We don’t have separate racks or separate switches – we just tell them ports and locations and that’s it. The devices are on a VLAN, all the ports are locked down. Conceptually, they look after the motorway and we provide the cars,” he explains.
“On the installation side, our techs pull in cables from doors to patch panels in the hub rooms, test them and verify them. The airport IT guys then come along and connect them.”
“We power our door controller/readers from a central location in remote IT hubs which also house the network switches. At the emergency door release, we cut in to control lock power which is fed back to the old Honeywell system and we do a permanent connection then we switch at the EDR.”
Driving the system
Browne turns to his AC2000 alarm and event display (AED workstation) and we poke around CEM Systems’ AC2000 management software. We start with the personnel page of a cardholder. It’s similar to others – showing details like personal information, access levels and card design and print options.
Next, Browne opens the monitoring screen which shows what the operators are looking at live. The layout is simple, an aeri
al image is used as a site map and dominates the middle and right hand of the screen but there’s a stack of functionality built in around the edges. Under the map is the event log comprising alarm type, location, status. Click on an alarm and it comes up on the left of screen in detail showing the action taken.
“One of things to take into account looking at this screen is that the old Honeywell system could not distinguish between a valid transaction and an alarm so any time there is an event an alarm comes up on the Honeywell screen,” he explains.
“The first call we got after switching over the DTB from Honeywell to CEM was security ops telling us there was something wrong with the access control system because there were only 4 alarms in the alarm log. We had to change the mindset of security staff here – the CEM system only tells operators there if there is an actual alarm.
“We had to explain they didn’t need to know if someone accesses a door – let the system manage authorised access – that’s what it’s designed to do. All you need to see are alarm events.”
While the CEM solution is now handling the DTB, the ITB is still managed by the Honeywell system. With 6 ADT techs working on the site, the plan is to cut the bulk of the ITB over around October. To this end, the team has installed CEM readers side-by-side with the existing system readers and the site is running double cards – the legacy Indala card and new DESfire CEM cards.
“The way we are doing the installation allows us to do all our commissioning – test all the time zones, all the functions on all the doors – IT allocates the ports – everything except the locks,” Browne says. “Once Airport signs off on the new system and we are ready to go. All we do is switch the lock power.”
Paddy Browne, engineer, Tyco
Watching Browne steering the system, I can see why CEM Systems has so many airports – the fundamental nature of this solution is so right for an application like this one – it’s operating at a whole different level.
Walking the site
Next we take a tour around the domestic terminal – it’s a huge facility even though the DTB is only 15 per cent of the overall size of the site. As we walk, Roger Read, project manager, ADT Security, tells me the project has been a learning curve for the team and the airport. Read says talking to stakeholders has been the biggest part of the project from his perspective.
“Within an airport there are so many organisations – no one person is absolutely responsible – you can’t lock people in a room and get them to bang their heads together till they work out issues – you have to bring them with you,” he says. “Implementing the system on a site like this would be challenging enough but all these other factors combine to make it more difficult.”
To illustrate Read’s point at one sliding entry Browne tells me that 4 different stakeholders told the team what the door functionality had to be based on different security demands.
“Something else that’s been interesting with the DTB for us is that they’ve been refurbishing this building as we’ve been installing the new system,” Read tells me as we thread our way through a bustle of travellers. “That means we’ve done the existing switchover, while at the same time installing new doors as the refurbishing process has gone along.”
We have a look at a CEM reader on an air bridge. Browne explains that the CEM system allows both individual access for cardholders or timed passenger mode access.
“The way it works is that after a set time – say it’s 10 minutes, the hold backs will release the doors and they will go back to being fully access controlled,” he says. “There’s an audible alert allowing airline staff to reset the counter if all passengers are not off the plane. In this mode the door will alarm after 5 seconds if it’s hooked into a hold back.”
Our next port of call is a hub room in a quiet part of the terminal. It’s a typical network room – bare walls, overhead cable trays, racks. In here are the controllers for the existing Honeywell system and this is where the patch panels and switches providing ports for the CEM system are located. The different footprints of the 2 systems are apparent. The 16Amp power supply with 4 onboard batteries supporting up to 5 doors is the largest housing on the CEM side – there are no controllers.
“When we are finished, on the wall of every hub room will be a laminated sheet with the door locations on it and inside this housing there will be a system schematic showing every reader and every location,” Browne says.
“This means a new tech who comes along at 4am and does not know the site will have all the relevant documentation from any one of these controllers for the entire system – which door is fed off which power supply in which hub location and which port the door is in.
“Techs can also access the network from these hub rooms and get access to information they need for trouble shooting – CEM is a great system for a tech – you can get on and ping a connection so as to verify if the problem is the network or the reader controller.”
Roger Read (l) project manager, Tyco, with Paddy Browne in the Auckland Airport security control room
Finally, we have a look at the control room – it’s a good-sized space as these things go, well laid out, with supervisors and management to the rear and rows of workstations with many system screens. Browne points out the event window on the CEM workstation – it has just 2 alarms. I also see the escalation window on the supervisor’s desk.
Conclusion
This is a large site – it’s complex in its physicality, as well as in its operations. What’s going to challenge Auckland Airport even more over the next decade is enormous projected growth in passenger numbers – up 25 million by 2025. This growth will demand expansion of facilities, as well as expansion of the access control solution.
“The airport is evolving so there’s a lot of future work to do,” says Catterall. “One of the reasons we ultimately selected IP was that this airport changes constantly, particularly in the retail area with doors moving around, The fact doors can be plug and play – that was a selling point.
“ADT and CEM are already engaged on future work that follows current work. Work is happening in the baggage area in 2 phases, we are already modifying our system design to cope with that.”
It’s just a sprawling, rolling, evolving application on top of a challenging site that’s growing fast, I muse.
“Yeah, it is,” agrees Catterall. “And that’s the beauty of the solution we’ve implemented – we’re getting rid of the legacy problems and modernising and this decision is streamlining our ongoing upgrade and allowing us to think about things like the Park and Ride facility and IP CCTV integration, too.
“We are doing some blue sky work with CEM for the GPUs – we are putting something together using CEM Systems’ emerald readers and we are going to re-program the GPUs so they operate in a time and attendance mode allowing us to automatically bill an airline for energy.”
Listening to the story of Auckland Airport’s ongoing transition from traditional RS-485 to IP, it occurs to me the team are telling the story of the access control industry’s looming digital transition. Thinking about this, I come to the conclusion that the length of the process, the direct interaction with the integrator and manufacturer – these were key contributors to Auckland Airport’s benchmark access control solution.
Forced by circumstance to think hard about the future, Catterall and his team thought deep, stretching CEM’s powerful product to another level – demanding not just an IP-compatible access control system but a real creature of the digital age, with smarts in remote reader/controllers and virtual real time controllers and virtual database servers buttoned up in multiple secure locations in the Airport’s private cloud. In the future, all serious access solutions will look this way.
By John Adams
