City of Dreams Macau Dallmeier IP Solution
IF you were going to construct the toughest video surveillance challenge possible using current technology you’d start out with a casino. That’s because this is one application whose CCTV requirements are so firmly bolted to operation and revenue generation that failure of the surveillance system means closure of the business.
City of Dreams Macau – Having picked your application you’d make this a seriously big resort and casino with multiple gaming areas across multiple floors, some public and some private. To complicate installation and operation you’d break up the gaming areas with retail precincts, restaurants, bars and major hotels. Then you’d tip in people. Vast numbers of people – 40,000 visitors each day.
To ramp up the technical challenges you’d plug in system parameters like real time recording on all inputs at D1 resolution. Next you’d add cameras – lots of cameras. Say, 5000 of them, fixed and PTZ. And you’d insist that these cameras have clever features like remote global configuration and the ability to handle low light and strong backlight.
To hold all this video footage you’d want lots of storage – let’s say 1TB per input with buffering built in. Then you’d insist on a distributed architecture model with cameras and storage devices in the same areas but deliberately routed to different local ELV closets with individual in-rack UPS support to maximise redundancy.
And you’d demand camera sharing between security operations and surveillance and the ability to assign given cameras to secure viewing rooms for authorities like the police or gaming commission as well as to authorised managers anywhere on the site.
The integration
But that’s not all. You’d decide that a system this big needs neural capabilities to link camera views to a multiplicity of events and functions across the site. So you’d ask for layer upon layer of high and low level integration between your video surveillance solution and systems like alarms and access control, and PoS and gaming, with alerts and data built into the management system, all this taking place in real time.
To drive a system like this you’d need a very special cockpit. You’d want your operators working in a high performance surveillance operations centre that maximised their ability to work effectively. You’d want unlimited simultaneous replay with video and data coming in across multiple ports and intuitive operational management of the facility through a powerful GUI with exceptional interfaces. You’d want an integrated surveillance room in every sense of the word.
Planning the solution
If such a solution sounds like a pipe dream think again. This site and this system actually exist at City of Dreams on the Cotai in Macau, where local integrator Elixir and City of Dreams’ in-house Surveillance Technical & Operations team have built the world’s largest IP surveillance solution using Dallmeier IP video gear.
The system was supplied by Dallmeier International, a joint venture between C.R. Kennedy and Dallmeier Electronic, which provides support for Dallmeier installations in Asia, Australia and New Zealand. Technical support for this project came from Dallmeier International’s Hong Kong office as well as from Dallmeier Electronic in Germany.
The Australian connection to City of Dreams is multifarious, from part ownership of the site by Melco Crown Entertainment, the team-up between local CCTV supplier C.R. Kennedy and Dallmeier Electronic, to Aussie director of surveillance, Leroy Daniel, a Canberra boy with lengthy experience in the casino surveillance markets in Australia, Internationally and now Macau.
Daniel is an operations animal. He started his career in a manpower role in Canberra 20 years ago before joining the team at Canberra Casino and switching to security monitoring and then surveillance. Thanks to this grassroots entry point, Daniel exhibits a wall-to-wall knowledge of casino security/surveillance operations. If there’s anything he doesn’t comprehend about the casino surveillance function it doesn’t show and his professionalism and enthusiasm are infectious.
“We had a vision – we had an expectation of what the system would be and that vision had to fit in with the fact this resort and casino are a showpiece – everything about it is designed to be the most advanced solution possible”
“This is the 8th casino project I’ve been involved with,” Daniel explains. “Establishing the surveillance operation here at City of Dreams was a major challenge but it was also most enjoyable because we actually got a chance to be involved in making the key decisions and in designing our space to suit casino surveillance operations.
“Most of the time with a casino you either inherit a system or by the time the surveillance people are engaged you find yourself stuck with a solution that has been jointly designed by anyone but someone who has actually worked in casino surveillance,” Daniel says.
“And when the operations people come in and say: ‘No way that’s not going to meet our needs’, you face a massive challenge. You’ve really got to get in at the conception stage otherwise you don’t know what you’re going to get.
“We were not only involved at conception and with designing the system and the surveillance room, we also walked around with the integrator during installation and as a result we got almost everything we wanted. If I was to do it again there are only a few things I would change.”
According to Daniel, the serious planning stage for City of Dreams’ surveillance system began about 2 years ago – but he says people were already doing electrical planning and architectural design work on this project as long as 5 years ago.
“This was not a retrofit,” Daniel says. “The data network went in during the build and the components of the surveillance network which would run on a 10Gb fibre backbone were installed after that.
“We had a vision – we had an expectation of what the system would be and that vision had to fit in with the fact this resort and casino are a showpiece – everything about it is designed to be the most advanced solution possible.”
As Daniel explains things, the original project was the vision of Lawrence Ho, Co-Chairman and CEO of Melco Crown Entertainment Limited and James Packer, Co-Chairman of Melco Crown Entertainment Limited. Melco Crown Entertainment’s idea was a state-of-art modern resort and casino leveraging all the latest technology.
“As part of this it was decided that every system in the resort and casino should run on the backbone of the IT network – including surveillance,” Daniel says. “So our IP cameras are on Cat-6 cable and the cables from each camera go to a PoE network switch in one of more than 220 remote ELV network closets spread around the site.”
According to Daniel, this architecture has advantages and disadvantages. It means short cable runs in the field and greatly enhanced redundancy, as well as increased flexibility for expansion of the system and its viewing points but it also means technical teams have to support hundreds of locations across the property.
“Obviously, for our technical team there were serious demands and pressures leading into the opening of City of Dreams,” Daniel explains.
“With casino surveillance, as soon as you open you go operational and face all the usual casino issues from day one.”
According to Daniel, the surveillance system at City of Dreams has a role best defined as Protection on Investment – or POI.
“POI means our key focus with surveillance system is revenue retention and asset protection,” Daniel explains. “Additionally the system is used to meet the requirements and needs of multiple stakeholders requiring access to the CCTV system such as the local Gaming Commission, Police Authority, Corporate Investigations and our own Security Control Room.”
The site and the system
It’s inevitable when you’ve seen many large CCTV installations to become a little blasé about even these giants of the surveillance industry. But there’s something awe-inspiring about City of Dreams that demands respect right from the get-go. With an average of 37,000 visitors a day, this is an enormous operation.
Level One of the casino incorporates the public gaming area – and it’s big – stretching off as far as the eye can see. At 420,000 square feet it’s one of the largest gaming areas I’ve seen by several factors. There are approximately 520 gaming tables and approximately 1350 gaming machines at City of Dreams.
As we walk through the site we never pass the same spot twice – I am instantly and constantly lost. Daniel explains it would take days to walk around the entire site and it’s impossible to argue. This site is truly vast – and this is just front of house. Back of house is a labyrinthine world with its own dedicated services, restaurants and amenities, where City of Dreams’ 7000 workers bustle through their business.
“There is a comprehensive range of accommodation options at City of Dreams include Crown Towers offering approximately 300 guest rooms, Hard Rock Hotel offering approximately 300 guest rooms and Grand Hyatt Macau offering approximately 800 guest rooms,” Daniel explains.
“There’s private and public gaming, and the next phase of development at the resort yet to come – this part of the construction includes a Dragon-inspired theater production in the purpose-built Theater of Dream, a major new retail precinct and significant additional surveillance – several hundred supplementary cameras.”
With a site this large and complex it’s important to get a handle on the parameters of the system early on so let’s take a look at exactly what has been installed at City of Dreams. For a start, it’s worth noting that this is an end-to-end Dallmeier IP Video system.
“From our perspective – and taking into account the need to maintain and service the system, we just wanted one product, one solution – we didn’t want a mixed bag,” explains Daniel.
“When we saw Dallmeier’s cameras operating we were comfortable with their performance, functionality and with the fact there was virtually no latency. Obviously this is very important for a resort with casino and once we’d seen the system in operation we were comfortable that IP was a viable solution for us.”
City of Dreams Macau Dallmeier IP Solution
Dallmeier hardware installed at City of Dreams includes fixed IP cameras, PTZ IP cameras, network storage units (NSUs), the EDS-1 encoder, decoder and streamer, variodecoders (multi-split software decoders), external mass storage devices, time servers and VMC-1 keyboards.
Daniel says that with its fixed cameras, Dallmeier provides different types of vari-focal lenses which are suitable for indoor and outdoor applications where lighting requirements vary.
“With its Domera PTZ cameras, Dallmeier provides modular PTZ dome systems with 18x or 26x Zoom lenses – this spread is perfect for us inside or out.”
Meanwhile, Dallmeier’s regional manager Terrence Chang says Dallmeier’s IP cameras used at City of Dreams are Cam_inPIX.
“They use Pixim chips so every single pixel is like a camera iris,” he explains. “This provides a programmable level of functionality so it’s possible to control and adjust a whole group of cameras from the control room by sending instructions across the network to improve images. This capability was a big advantage at City of Dreams.”
The software driving the system is Dallmeier’s security management system, SeMSy, replete with a significant number of high and low level interfaces to systems including the CardaxFT access control solution. Other interfaces include external systems such as Angel Eye Card Shoes, Micros PoS, IGT Slot Machines and RFID-capable gaming chips.
“We have around 4500 IP cameras on the site with another 500 soon to be installed – that includes security and surveillance cameras,” Daniel says.
“The system covers front of house and back of house, and there’s the building above ground and significant parts of the building below ground as well as a network of back-of-house corridors – it’s a major infrastructure and a huge investment.”
Dataveillance
According to Daniel, there are no other CCTV applications that are as demanding as casino environments or have as many cameras per square metre as casinos do. As your eye roves around the ceilings at City of Dreams it catches on dozens of cameras scattered across the surface – yes you think – he’s right.
“Whether it be airports, hospitals or industrial sites – none have the concentration of cameras casinos must have nor the demand for real time monitoring on multiple stations, with playback of high quality footage going on left, right and centre,” Daniel says.
“The size of these systems brings unique challenges – you can’t effectively handle casino surveillance in the old way. Properties of this scale where you’re managing many thousands of camera inputs are way too large to rely solely on a process where operators are scrolling through endless camera views hoping to catch someone doing something.
“That’d be like playing darts with a blindfold on,” Daniel chuckles. “What we needed was to integrate the surveillance system into the core revenue streams of the business – we wanted to facilitate something known as ‘Dataveillance’, a term coined by U.S casino surveillance industry expert Darrin Hoke.
Just what is Dataveillance? As Daniel explains, it’s a level of integration and interfacing so complete it allows property wide “business intelligence” and historical data from real time systems driven directly into the surveillance operations centre that can be further analysed/actioned for trends or events of concern.
“Most surveillance systems might be able to interface with an access control system but we wanted to go further than this,” Daniel says. “What we got with Dallmeier was the ability to use its management system to drive high and low level interfaces linking all parts of our business.
“What this means is that our surveillance operations centre is tapped into point of sale for the bars, into the gaming tables like baccarat our core Gaming product, as well as being integrated into the slot machines – and there is even RFID in our gaming chips.
“Then there’s integration with alarms sensors, duress alarms and access points across the site. And it goes even deeper. With our baccarat games the system interface provides us with the results on-screen of the games so there can be no disputes.
“So what Dataveillance describes is this wealth of information/business intelligence coming in and being used to create automated alerts, alarms and events so operators know where to look at what time and when. Additionally collated/historical data can be disseminated, manipulated, exported or analysed for anomalies – and bear in mind all these are real time interfaces on thousands of cameras.”
Daniel says this sort of integration was one of the key reasons City of Dreams went with Dallmeier. He says that when City of Dreams management looked at other vendors’ products they saw that casinos were only a very small portion of their business.
“We knew from experience with such vendors that if you wanted a special software customisation or requested a new interface be developed, unless it had a direct impact on that vendor’s international sales then they might say ‘yes, we can deliver it’, but they wouldn’t do it. It’s the reverse with Dallmeier – their core market is casinos so these special interfaces are simply what they do. We are in the same business – and that’s important.”
Chang agrees the interfaces were an important part of the installation for Dallmeier.
“We were heavily involved in creating the correct software integrations for this installation and that was challenging,” he explains. “Our engineers needed to work out how to use the information provided by systems and to decide how to display the data gathered by integrating these systems onto the surveillance system’s screens.”
The installation
Daniel explains that CCTV is one of the last things that goes in during the construction process and this added to the pressure on installers early in the year as the job progressed.
“There’s a line of dependency with the construction process that means the timeframe for the system integrators and installers always gets compressed,” Daniel says. “In practise our timeline got smaller and smaller and our buffer zone got compressed.”
At City of Dreams, the hardware side of the video surveillance solution was installed and commissioned by a team-up of local system integrator Elixir and surveillance technicians, in-house IT engineers and supported by Dallmeier.
“We’ve got an exceptional in-house technical team that helped install and that will maintain the system – we also have a service level agreement with our integrator, Elixir,” says Daniel.
“They have the intimate knowledge of the data network and the ceiling layout. As well as the physical installation, a lot of the work with this system took place at the backend – there was an enormous amount of configuration required.”
According to Terence Chang, Elixir had more than 30 technicians working on the physical installation.
“Elixir had several projects during the build stage of City of Dreams including the data network infrastructure, the AV solution and the surveillance system,” Chang explains.
“Most surveillance systems might be able to interface with an access control system but we wanted to go further than this,” Daniel says. “What we got with Dallmeier was the ability to use its management system to drive high and low level interfaces linking all parts of our business”
“As well as the techs there were about 10 engineers working on the system, commissioning, troubleshooting, training and helping the installers.”
“Dallmeier was involved from a very early stage – having proposed our system we needed to be aware of the planned network structure. We started thinking about the sorts of products that were coming up for release and which cameras would be best for the site,” says Chang.
The challenges of the surveillance installation are readily apparent as we move through City of Dreams. Something else that is equally clear is that it would have been impossible to build a solution like this using analogue or hybrid technology. The site is so complex it would have demanded multiple control rooms and a far larger surveillance team while resulting in serious operational deficiencies.
As well as the challenges of installing and integrating such a large networked solution, the City of Dreams installation team also faced the sort physical challenges you’d expect in a site like this, where design and functionality must be brought together in an ideal compromise.
“Casino CCTV in particular has several issues to contend with. First there are the normal ceiling design issues which relate to the presence of air conditioning hardware, lighting fixtures, AV, background speakers and fire systems,” Daniel says.
“Most people only ever have to deal with flush ceilings or pop-out ceiling tiles but we faced a challenge with City of Dreams’ complicated ceilings,” he explains. “When you look around this property, it’s different – design is a key factor – the designers wanted to make a statement.
“In terms of the physical installation there are layers of complexity. We have large amounts of fibreglass moulding and we also have chandeliers and other lighting fixtures. Then there are decorative lighting wings that include reverse LEDs putting high intensity lux back onto the ceiling.
“Combine all these and you can see it was a big challenge for us camera-wise and as always, many of the problem areas were right above the gaming tables – exactly where we needed our cameras to be.”
Daniel says that getting through the design and installation process which involved consultants, designers and architects was challenging.
“We had a few debates in the course of the design and commissioning and we came up with some compromises that balanced functionality and aesthetics. To be honest, one of the key investments we make with CCTV is in the protection of revenue streams coming in across the table so if we can’t see it we can’t protect it – it’s that simple.”
Daniel says in some of the chandelier rings the installers integrated the cameras into the bottom of the ring – cabling and power for the lighting and Cat-5 for the cameras comes down through the struts of the chandeliers. Installers used phantom struts in locations where existing struts did not suit.
“In order to get below the ceiling we also used some ceiling droppers and for some of our PTZ cameras that need 360-degree vision we had to drop the cameras below all possible obstructions. Those droppers were all locally manufactured.
“We use mostly PTZs at City of Dreams,” Daniel says. “This is partly because the Macau casino market is dynamic and we and other casinos put in as many PTZ cameras as we can so we can change the gaming floor layouts depending on visitor trends. With PTZs you can move tables and re-set PTZ cameras and lock new presets in, minimizing the down time.
“There are also benefits when it comes to commissioning the cameras and setting them up – with PTZ cameras you can cut the required time by around 75 per cent.”
According to Daniel, complicating matters at City of Dreams was the fact the casino has different types of chandeliers and different types of droppers were needed to suit each type of chandelier. The difficulties change shape as we move around the site. In one area there’s a lattice ceiling and this needed a different type of solution altogether.
According to Daniel, as per our requirements installers put in 3 cameras per table with 2 cameras in the ceiling and a camera in each table’s display to give an inspector’s eye view of games to resolve player disputes.
“When disputes arise this is the camera that allows us to ascertain exactly what happened,” Daniel says. “Pit management has restricted access to these cameras through the system at the pit stands and they can review back and we can control the amount of back time they can view up to and we also receive an alert when in use. Doing this is easy thanks to the networked Dallmeier solution and we know who is looking at what footage and when.
“That enables us to push down low level disputes to the gaming floor and free up operators to concentrate on more serious matters. It improves customer service and makes resolution quicker. What we’ve done is given a bit away downstairs without losing visibility upstairs.”
According to Chang, this is typical customisation of the Dallmeier system.
“This particular capability is not available to anyone else – because City of Dreams wanted viewing of the gaming pit we specially developed this and customised it as to how long they wanted pit management to be able to view and control the cameras,” he explains.
Next, we head up to level 2 – the VIP level and here Daniel says that the building’s design and architecture presented installers with additional challenges.
“The plan was for a wavey ceiling design so we had to get a customised option to have camera, backbox and housing installed in this undulating environment and yet have the camera horizontal.
“It sounds matter of fact but it was not,” Daniel explains. “There was a lot of locating and relocating and patching and matching holes, and a lot of painting – these multiple ceiling heights and designs were tough. Using mostly PTZs was a big advantage, especially seeing these could be configured remotely.
“A major issue we had to contend with in these small VIP salons was lighting – in one VIP area there are crystal chandeliers and installing the cameras into the fittings was not the only challenge,” Daniel explains.
“The effect of these fancy fittings is side light and in order to cover the traffic flow we needed our cameras in front of the chandeliers. There was some difficulty with this as dropping cameras in front of the lights would have impacted on aesthetics. We decided to go straight up the middle.”
Daniel says as a general rule, lighting is one of the biggest challenges in casino surveillance and City of Dreams was no exception.
“Some tables will have no lighting, some tables will have too much lighting or spots of strong lighting. We were scientific in our original planning for lighting – we took 6 reference points across each table and made sure we didn’t have greater disparity than 50 lux between each of those reference points.
“If you have lighting hotspots on the tables you get glare from the cards and chips coming back to the camera – we were lucky we were able to drive this – we’ve got really good lighting on tables and at cash points.
“One thing that is interesting is that as part of the commissioning process we turned the lighting up as far as we could then down as far as we could and we found that even with our minimum emergency lighting on the tables we can still see the cards and chips,” Daniel explains.
“And most casinos have to close their tables under emergency lighting conditions but we could continue operating – we put that down to the quality of Dallmeier’s IP cameras.”
Daniel says all the cameras at City of Dreams have smoked domes.
“Some other casinos have gold or chrome coloured domes but when you go with that you lose another F-stop so we decided to stick with our smoked domes.”
Chang agrees smoked domes were the right option.
“The Dallmeier IP cameras work well with the smoked domes in internal environments,” he explains.
You’d expect illumination and not just light fittings to have been a major part of the installation but according to Daniel, that wasn’t the case and with good reason.
“We put in no additional lighting at all,” he says. “While we do have specialised lighting requirements for tables and cash handling areas, this was part of the original plan so there was no need to retrofit lighting for these areas. The combination of the Dallmeier cameras and our original specifications means we have excellent performance with the installed lights.”
The network
At City of Dreams the network is built using a combination of Dallmeier hardware and high-end Cisco network devices. The design of the network is based on a service-oriented network architecture with 3 layers (Core, Distribution and Access layer) with layer 3 routing on multiple subnets.
In the ELV closets NSUs in racks are equipped with 2 hard disks and handle recording in parallel (Raid-1) for redundancy. The quality of stored images is D1 (720 x 576 pixels PAL) with a frame rate of 25fps using an MPEG 2 compression protocol. Additional long term video record archiving is being achieved via SAN storage. Hard drives used at City of Dreams are from Western Digital and they have a total storage capacity of around 5000TB – that’s a massive 5 Petabytes.
In terms of physical installation, each IP camera is connected with an NSU on the same VLAN at the same edge switch. In this way, the system retains the network traffic locally at the edge switch instead of routing to other switches via distribution and core switches – Daniel says this is an important way to control bandwidth. Another installation challenge related to the network was ensuring the surveillance signals were in a secure environment.
“There were security issues with sharing the data network – this was one of the key challenges for Dallmeier and Elixer and for our IT guys,” Daniel explains. “We needed to provide a surveillance network that had a demarcation between the data network.”
There was also concern for the integrity of the network.
“There have been cases in converged networks where upgrades have caused the surveillance system to shut down so we had to manage this,” explains Daniel. “We could not afford to have an IT tech to flick a switch on an upgrade and turn our surveillance system off or impact its performance.”
Meanwhile, Chang says the layout of the system is robust and capable.
“Each Dallmeier server is capable of handling 1500 devices at a time – that’s 3000 cameras on 2 servers,” he explains. “With this system you don’t have to wire each input to a central matrix – you just run cable to a nearby network switch – it’s that simple.
“If the main server fails the system fails over to a standby server and if that goes down the workstation can still receive live footage and play image streams back – that’s the beauty of the system.”
Daniel says that during the latter part of the process of installing the system the team went from considering a layer 2 option to selecting a layer 3 option.
“That was a last minute change,” Daniel says. “The team thought a layer 3 option would offer a more resilient system. If we’re monitoring an incident and sequencing through multiple cameras, we’ve got to have continuity of footage to make the video stream admissible in court. You don’t want the system to blank screen while switching between cameras and that’s what layer 3 gives us – better and more reliable performance.
“The direct management of the video surveillance system is handled by our surveillance system IT architect and the technical team,” Daniel says.
“However, the City of Dreams IT department is heavily involved with the network part of the system as well as connecting to external systems in order to facilitate real time high level interfaces to core systems.”
“The design of the network is based on service-oriented network architecture with 3 layers (Core, Distribution and Access layer) with layer 3 routing on multiple subnets. In the ELV closets NSUs in racks are equipped with 2 hard disks and handle recording in parallel (Raid-1) for redundancy”
Daniel explains that on the installation side the technical team was headed up by surveillance technical manager Kevin Iek, surveillance IT systems architect, Roberto Leong and surveillance assistant technical manager, John Hong.
“These 3 really drove the project,” says Daniel. “They worked 7 days a week and 18 hours a day to meet the deadline and they are still working long hours expanding and customising the solution to meet the casino’s changing needs.
“These guys are the brains behind our system. When we opened City of Dreams we knew we were going to go IP and we knew to do it we needed the best IP video people in Macau – we were lucky to find these three gentlemen.”
According to Iek, it took 6 months to build the surveillance system at City of Dreams.
“There were some challenges for us – the first one being the ceiling structure,” Iek says. “It’s very complicated and involved installing some cameras in the lighting – there were over ten different mounting options required. The Dallmeier IP cameras easily integrated into all our required mounting options without any performance or overheating issues.”
Iek says that network’s backbone is more than 10Gb – based on the architecture there’s 3-layers and he says with current camera numbers it’s not possible to get the network load up to 10Gb.
Meanwhile Leong says the technical team worked closely with the IT department, especially where the network interface was concerned.
“We were focused on the interfaces and ensuring these worked efficiently – the interface with the PoS system for example,” he explains. “So far we have not encountered any problems with this networked solution – the system has been very reliable.
As Daniel explains, an important part of the installation was to drive the vendor to vendor interfaces and agree on protocols and software.
“This involved agreeing on which program to use and working out who is pushing and who is pulling the data,” he says.
“As you can imagine, there were a lot of lists created at this juncture of the installation – it’s an art to drive the process and to get things working as requested while meeting operational requests and the technical team did a brilliant job.”
In terms of architecture, Leong says the system is separated and there are individual VLANs carrying PoS and video signals.
“The network is divided into the gaming network and the non-gaming network and there are firewalls to separate the 2 – we have local firewalls as well – it’s a closed system with no external access.”
Meanwhile Hong says that expanding the system is very easy.
“Compared to the analogue systems where you have limited inputs and outputs IP allows as many cameras as you can fit on the network – there is no limit as long as the network can support them,” he says.
The Dallmeier system uses MPEG-2 compression with cameras recorded live on all inputs 24×7.
According to Iek, some of the benefits of the Dallmeier system include the simple installation of fixed cameras.
“You lay one cable and that gives you power, data and control,” he says. “You do need a separate power cable for PTZ cameras but both fixed and PTZ cameras can be configured over the network. With the Dallmeier cameras there are multi settings specially designed for casinos.
“We can lock onto presets to get the best pictures very quickly. We still have to do backfocus but we can fine-tune the camera remotely so it’s much easier and more flexible than analogue. To make changes we just define the IP range and then just send the changes out across the network.”
“One of the benefits of the layout is that each camera has 2 hard drives in the NSU and there is a pre-disk buffer so we have no downtime even if there is HDD failure. The cameras multi-cast and on playback images come straight remote NSU.”
“IP is ready,” says Leong. “It’s more powerful and more flexible than analogue. In the past we’d have to lay a coaxial cable from the camera to the matrix for every input which is very hard on a large site. But this is just Cat-5 to a nearby ELV closet – it makes the system very flexible and much easier to install. The only way to build a system like this is using IP.”
Security control room
Like most resorts and casinos, City of Dreams Macau has separate control rooms for surveillance and security. The security room at City of Dreams is an impressive set-up in its own right.
“This room has core functions, including to support the security operation but primarily its job is alarm management and the team here is looking at where the alarms are going off around the property, where is the integrity of the perimeter being breached – where are doors being left open, duress alarms being activated – anything related to the physical security of the business,” Daniel explains.
“There are many thousands of alarm points. There are about 1,000 access control readers linked to the system – it’s a layered approach where to get from a front of house to a back of house area you have to actually swipe a card. The security room controls the lifts as well.”
Importantly at City of Dreams there’s a good understanding between the security and surveillance functions in part thanks to the close relationship between Daniel and fellow Australian and leader of the Division, City of Dreams Vice President, Security and Surveillance, David Mackay.
These 2 have the sort of effortless working relationship that goes a long way to explaining the unusually close synergy between the surveillance and security functions at City of Dreams.
Mackay explains that the Security and Surveillance operating model is a very modern approach to resort asset and revenue protection. “It is different to the way most others do it but it optimises protection for the business. Many of our competitors run models that require Surveillance to monitor just about everything. In my view this comes at the expense of monitoring the highest risk and core revenue areas. Why get Surveillance people with game protection training and knowledge to focus on more petty issues? It isn’t good business”, says Mackay.
“Basically we allow the security room to look at certain cameras on the gaming floor as well, which is a real departure from traditional casino operations,” Mackay explains. “We are particularly selective in allocating cameras to ensure the integrity of Surveillance monitoring is not compromised. Our Security Control room receives particular cameras to enable them to be effective but they don’t get them all.
“Importantly, the security team does not get the table-focused cameras – that’s not allowed however they do get the general pedestrian walk areas so they can follow people moving through the casino. Obviously the surveillance room monitors all the gaming tables – they have a higher level of training in gaming and game protection which is appropriate for their role.”
“By allowing the security team to take care of undesirable activity around the complex including on the gaming floor it allows the surveillance team to concentrate on the gaming tables,” Mackay explains.
“The way it works here is that the security room has access to security cameras for physical security purposes while the surveillance room’s cameras relate to gaming tables and machines – there’s a demarcation between security and gaming and it’s important we keep them separate for a number of reasons but they support each other,” Daniel explains.
And according to Daniel, it’s the Dallmeier IP system that allows such easy sharing of tasks between 2 separate monitoring locations.
“With digital systems it’s easy to manage and monitor the allocation of cameras and to make sure that only the allotted cameras are being viewed by the security department,” he says. “This delineation is important.
It’s against the rules for security to view the tables and with the Dallmeier system this divide is very easy to control and to monitor.”
Daniel says the traditional casino surveillance model was that surveillance had all the cameras inside and security had all the cameras outside.
“But having started my career in security and then moved to security control rooms and then to surveillance, I can see the needs of the different areas as well as the operational weaknesses of the old model,” Daniel says.
Mackay agrees.
“From my perspective, the key thing is to maintain the integrity of the property’s operations in any way possible,” he says. “We make sure that through a process of assessment and analysis of need versus risk that we don’t compromise protection. We don’t want to operate an old model when it isn’t optimal just for the sake of Surveillance old school traditions.”
Mackay says given that the security room has to support the security operation, alarm systems, back of house, perimeter, deployment of staff, radio systems, lift monitoring, fire systems and more; then it’s a logical step for the room to also support the security element that concerns CCTV as well.
“It optimises the security function this way and leaves Surveillance to focus on our core revenue streams.”
The Surveillance Room
City of Dreams’ surveillance room is the centrepiece of this installation and it’s an unforgettable sight. More than 200 large screens are arrayed in concentric circles around a central point like a liquid crystal Stonehenge. It’s unlike any other surveillance room in the industry. Oval in shape, the management position of the City of Dreams surveillance room is a raised ‘bridge’ in the centre of the room with operator stations encircling it in outward facing tiers.
It’s an unusual layout, yet it’s perfect. As you move around the room you appreciate more and more the fact this layout allows management on the bridge to scan the work of any of the 30 or more operators at their multi-screen consoles by simply lifting their eyes.
“We put a lot of thought into our surveillance operations centre environment,” says Daniel. “Most surveillance rooms are set up in a similar way to security rooms with wall monitors up the front, operators in front of them and management up the back. This limits management’s vision of what is happening in the room to the back row of screens.
“The idea here at City of Dreams is 360-degree communication and visibility – it allows management to see all the monitors from a central location – we have core management embedded in the heart of the operation and there are layers of management on lower tiers providing hands-on support.
“These senior operators and supervisors on the next level down are constantly going around the room behind the operators guiding them with tasks, coaching and undertaking performance management.”
Daniel says something else that was important was putting ergonomics into the operator console stations.
“Ergonomics, comfort and usability are all big parts of this room,” he explains. “Operators can raise their workstations and stand at the consoles to get the blood flowing.”
Daniel says that each console has dedicated task lighting which operators can adjust and he says the room lighting is also dimmable.
“Surveillance achieves results through the eyes of the operators so it’s important operators are completely focused and have the best of everything,” he explains.
According to Daniel, the surveillance operations centre is divided up into teams, the southern dragons, northern dragons, eastern dragons and western dragons. Asian themed and derived from the totally immersive multi-media experience Dragon’s Treasure showcased in The Bubble, a dome-shaped theatre and one of the iconic landmarks located in City of Dreams.
“Each of the sections has specific responsibilities relating to the main floor or the VIP salons, there’s an investigations and analytics team and there’s a dedicated team for point of sale, slot machines and card shuffle rooms, cage/count and security.
Daniel says that while some casinos have monitors on a wall City of Dreams brings these down to the consoles.
“The operators have working monitors and peripheral monitors – they can view automated alarms through high and low level interfaces, they can bring up a GUI – we have camera numbers but it’s hard to remember all these – instead with SyMSe there’s a GUI map and if you click on a camera icon it will bring up the camera, so for tracking someone through the property or for getting reviews it’s fantastic.”
“If we are reviewing an event we can use the map to track an offender back through the resort and casino – it’s very useful. When you compare it to the old VCR days it makes me wonder how we did it.
“The VMC-1 Eagle keyboards driving SyMSe are excellent, too.” Daniel says. “An operator with a jog shuttle can be monitoring someone live, press one button and start doing rewind immediately on their console monitor – it’s a big advantage – it’s intuitive.
“It’s just as easy to review events at point of sale,” he says. “We have an on-screen display that tells us the details of every transaction – we can filter out no-sales and void transactions and because they are all bookmarked in the Dallmeier system we can click and play and see what has taken place.
“The system is multicasting so any number of operators can be viewing scenes at a given time,” he says. “There was a lot of performance testing before we opened. We set the system up to view live on 9 split screens on every monitor along with synchronised playback. In fact, we tested the system under full load and it passed with flying colours.”
Daniel says a key issue to address in the surveillance room was latency.
“In casino surveillance it’s important to have no latency – I’ve found in the past that if there are latency issues the operators will change the way they work – they will zoom out and use overview scenes and avoid changing the camera views – but we want them to focus on the detail.
“The latency on our Dallmeier system is negligible and to be honest I’m very impressed – every time you view a camera the network has to link through multiple switchers and there is a compression process to go through and I would expect there to be some noticeable delay but there is none.”
It’s impossible to talk about the surveillance room without mentioning the level of training operators must go through to reach competency. Daniel explains that a typical casino dealer may learn 2 major games and one minor game.
“But surveillance operators have to learn every game in the house and not just learn how to play them but how to cheat when playing them and how to advantage play,” he says. “Then operators have to learn gaming procedures, security surveillance procedures, cage procedures, Food and Beverage procedures, and slots procedures.”
“Once they finish all this they have to learn the procedures of every other single department and division in the resort and casino. Then they have to learn how to manage all the numerous on-line monitoring systems we use. Just the basic training period lasts 3 months and it takes around a year to train a good operator.
“IP is ready. It’s more powerful and more flexible than analogue. In the past we’d have to lay a coaxial cable from the camera to the matrix for every input which is very hard on a large site. But this is just Cat-5 to a nearby ELV closet – it makes the system very flexible and much easier to install”
“This training is ongoing – operators are always learning,” Daniel explains. “Importantly, we can replicate any possible gaming event here in the training rooms in the surveillance centre – we have all the same equipment so we can break problems down in a test lab environment– if there are new techniques that come out to beat the casino we can break them down and work out the best ways to use cameras to spot them.”
Daniel says that in order to pass the course operators have to get at least 95 per cent in their final exams.
“Our operators at City of Dreams averaged 97 per cent.”
Conclusion
According to Terence Chang it’s a big decision to switch from an analogue or a hybrid solution to a networked solution but he says the installation at City of Dreams has worked just as planned with no major problems.
“Our IP solution is very mature now and capable of significant integration and scaling,” Chang explains. “One of the things about the installation at this site is that it is always changing, more cameras, different locations – only the flexibility of IP can support such a demanding application.”
Meanwhile, Mackay says the team at City of Dreams has had great support from senior management.
“We are very lucky that the company takes security and surveillance seriously and chooses to invest in and support our capability with a high quality system,” Mackay says. “The technologies in this room make the operators more effective – instead of wondering ‘which table do I monitor’, the team receive and can see information and deviations from what they might expect to see as wins and losses on tables. This makes the team and this room so much more effective – we’re very happy with that and our senior executive can take greater comfort in knowing we are protecting the business while saving labour resources in the process because of the initial investment”.
“In some respects City of Dreams works the way it does because we have a clean slate and this is a new team in a new facility, but there is some element of legacy with this site because we did set up Altira Macau (formerly Crown Macau) and there’s a continuation of that culture here,” Mackay says.
“We also hired locally and that has absolutely paid dividends. Around 80 per cent of our workforce is local Macau people and we have a solid mindset to develop all and promote from within. We are very happy that the foundational elements that reward great performance and develop local people came across to this new site and have become part of our road map and plan for the future.”
Daniel too, is pleased with the results.
“We were very fortunate with this project considering the size and scope and the issues and challenges one may expect,” he says. “Between the Elixir team, the Dallmeier team and our internal technical people, we got delivery on the system basically according to schedule so we had time to fine-tune and test bed the system before opening.”
There are those who say IP is not ready for the big time when it comes to video surveillance but Daniel strongly disagrees with this assessment based on direct experience at City of Dreams.
“We are living proof that large scale IP surveillance solutions are a viable option for the CCTV industry and can provide capabilities far above those offered by analogue or hybrid solutions,” he says.
“With the proper network design, it’s possible to minimize traffic across distribution and core switches to meet necessary performance parameters,” Daniel explains.
“I have no doubt IP cameras and solutions are the direction the industry will be heading. Apart from operational benefits to the business, it also presents a more cost effective solution for new projects by leveraging necessary IT infrastructure.
“We get a lot of surveillance executives from around the world coming to have a look at this system. People say it’s the best surveillance room they’ve ever seen, the design, the functionality, the equipment and the tools,” Daniel says. “And with more than 34 consoles active it’s the largest 100 percent end to end IP surveillance system in the world.”
Daniel thinks that if there is a defining quality of the IP surveillance system at City of Dreams it’s the ability of the Dallmeier system to interface the surveillance system with the City of Dreams business.
“Integration into our security, gaming, cash and revenue systems gives real time alerts, data and business intelligence direct to the surveillance operators allowing us to adjust and focus our resources according to the immediate business demands at the time,” Daniel says.
“This differentiates ours from any standalone CCTV system where operators are simply punching through cameras hoping to stumble across something – a technique that with several thousand cameras is not a serious option.”
According to Daniel, casinos have a unique operating model and Dallmeier was able to meet the demands of this application.
“We needed to be able to customise the system in the surveillance operations centre so it conformed to our operational model and Dallmeier was able to turn on a dime and to program and re-program on the run in order to deliver us the functionality we needed,” Daniel says.
“There’s been very strong support from Dallmeier – a whole team of Dallmeier people came over from Germany during the commissioning and they are still coming over to check and re-check the system. They were right there with us throughout commissioning and training. Dallmeier has offered us unconditional support for our IP video system forever – that attitude gives us a lot of confidence.”
Could this system have been built using analogue or hybrid? Daniel thinks not.
“Honestly, we could not have built our surveillance system any other way – it works beautifully,” he says. “I think a lot of people are misinformed about IP. But when they see our surveillance operations centre they quickly change their minds.”
“There’s been very strong support from Dallmeier – a whole team of Dallmeier people came over from Germany during the commissioning and they are still coming over to check and re-check the system. They were right there with us throughout commissioning and training”
More SEN News here.
