ANALYTICS as a technology seems to be on the brink of a momentous acceleration, the impact of which would be profound. The latest symptoms don’t simply identify faces, vehicles and other parameters of movement, they can detect gender, clothing, hair colour and facial expressions, too.
VIDEO surveillance, for all its capacity to inform security staff of developing events, has limitations. It can only take a proactive role when control room operators are watching footage of a camera viewing an incident in real time. In larger systems, no one has the time to watch hundreds, even thousands of cameras, meaning even the latest IP CCTV systems fill a proactive role of assisting investigators after an incident has taken place. Make no mistake, a digital surveillance system with the latest clever search functions is a joy to use – the ability to flit along a timeline makes searches extremely powerful but it doesn’t make these systems proactive – not in the way the one day could be.
What might analytics deliver a security team? It depends on the application, but you’d be thinking about early warning of events that breach programmed thresholds – that leaves a lot of wiggle room if the analytics are discerning enough and the video streams are of sufficient quality. You might detect a known license plate, a known face, a movement against the flow of foot or vehicle traffic, a vehicle where it should not be, a vehicle moving too quickly in a controlled area. With the latest analytics solutions, developers are talking about recognition based on gender, clothing colour, hair colour, age – even mood, though this might be a grey area, particularly on Monday mornings.
According to Will Hasna of Bosch Security, we have reached a tipping point when it comes to the application of intelligent video analytics to surveillance solutions
“Definitely yes,” he says. “Artificial intelligence (AI), machine learning (ML), deep learning (DL) are buzz words these days, driven by global technology companies like Google, Apple, Intel, etc. “With this trend continuing into the future, consumers will want more intelligent devices and that will influence the development of the security surveillance market as well. The miniaturization of chips and embedding of GPU and VPU inside devices makes the possibility of implementing AI, ML and DL into NVRs and even cameras very possible. That will revolutionise the use of video analytics by taking it to the next level.”
Hasna says the most commonly used IVA functionality has always been the basic event alerts such as line crossing, object in field, object left/removed and counting, these are the most common detectors available in the market and these are easily understood by the end user.
“However, when it comes to defining the most useful pieces of IVA functionality, each of these detectors used in isolation does not brings much value to the user,” he argues. “The ability to combine multiple types of detectors to customize to the needs of the user is the key factor to successfully utilising IVA for surveillance.
“Bosch IVA can combine different combinations of detections in a logic before an event is triggered, as an example, we can classify that an object has to be a car, that crosses a particular line in a certain direction before stopping at a particular area for a duration of time and only when all conditions are met will the camera generate an IVA alarm. You can even schedule different IVA profiles (conditions) for the day and use another type of IVA for the night.
“One functionality of IVA that is least used but in my opinion is the most important in functionality is the ability to conduct a Forensic Search. In the case of Bosch, we embed IVA metadata generated from the camera into our video recordings, which allows a user to define ad-hoc IVA rules/logic <I>after<I> the fact to search through the recordings. In most applications, the user does not really know what will need to be detected until an incident occurs.
“If using the Bosch Video Management System (BVMS) in combination with Bosch cameras, a user can simply select Forensic Search, and retrospectively select the following rules as an example, “Cross the line” combined with “Object classified as a car”, that “Matches the colour blue” and is “Travelling in a certain direction” and the BVMS system will return all the relevant matches. This gives the user the ability to search for actionable evidence quickly.”
When it comes to the best options between in-camera or in-software analytics, Hasna says there’s no best option for every application – both solutions have pros and cons.
“In-camera IVA provides speedy detection with minimum hardware needed to deploy and also helps to reduce bandwidth utilisation as the processing is happening at the edge, however, due to the limitations within the camera CPU, sophisticated algorithms such as facial recognition can’t easily be deployed without the need for additional server support,” he says.
“In-software is the opposite of in-camera. It has the CPU processing power to run complex detection, but because of the complexity of the algorithms, generally is only able to support a small number of cameras, requiring more servers to support more cameras. It also means introducing a single point of failure that will affect multiple cameras should the server fail. In my opinion, best practise would be to balance the deployment of in-camera and in-software depending on your needs. Using the in-camera to detect events that need immediate attention and using in-software to complement the in-camera lack of CPU power, such as facial recognition, license plate recognition, etc.”
When it comes to the most common applications of IVA, Hasna says that while there’s an increase in interest, analytics is not commonly used and when it is deployed, it’s often used to handle very simple tasks.
“The implementation rate in Australia is still relatively low when compared to the number of cameras deployed,” he says. “We see the most common functions being counting, where the user needs to keep a count of how many cars or people have passed through certain areas, as well as directional follow to ensure objects are moving through the intended direction.
“Bosch being heavily involved in the automotive sector providing may sensor technologies, including video, to vehicle manufacturers and with the heavy investment of these companies into autonomous driving technology to improve their detection systems is being leveraged to improve the analytic performance of the algorithms in Bosch cameras.
“Bosch analytics profiles have been developed over a long period of time and believe there is no one type of IVA that works best. The accuracy of IVA depends very much on many factors such as, what a customer is operationally trying to achieve, the scene itself, lighting conditions, or even the amount of people moving in the area as just some examples.”
There are pitfalls/limitations when it comes to IVA applications in the real world and Hasna says there are some things integrators and end users should bear in mind. Contrast between the object and the background is the most important factor, Hasna argues.
“Having a high contrast between the background and the object will increase the detection reliability. The smaller the size of the object the harder for video analytics to detect. As mentioned earlier, cameras have limited CPU power, the number of objects a camera can detect and track simultaneously or how accurate it can detect, is subjected to how well the algorithm is written and how powerful the CPU in the camera is.
“There are a lot of cameras available in the market place claiming to deliver IVA at the edge, but not all are created equal as an example it is generally easier to have accurate analytics in a controlled indoor environment that when the same camera is placed in an outdoor environment looking at a wide seen. We recommend that integrators/end users perform a proof of concept on the camera they have selected in the environment they would like to utilise IVA to evaluate the actual performance is going to meet their expectations.”
Is it possible surveillance solutions will become virtually automated – calling in human response only when necessary? Hasna thinks not.
“This is unlikely to happens in near future, AI technology is still in its infant stages – much more time is needed before such automated systems can be deployed safely,” he says. “Currently IVA serves as a virtual security guard watching over a scene 24/7 and alerts to the human abnormalities in the scene, which allows the human to make a judgement whether it is a false or real event and make a judgment. The task of IVA currently and in the future is to lighten the load of human response so that security operators can focus on the most important areas in the site.”
When it comes to integrating functionality suites such as retail analytics to increase return on investment Hasna believes the answer is yes and no.
“Retail analytics only provides a summary of the data gathered from these IOT devices,” he explains. “How the data gathered should be interpreted still depends on the retailer. Most retailers are not experienced or do not have the knowledge to understand such data. Not knowing what those graphs, heat maps or tables mean will not help the retailer improves their sales.
“On the other hand, the supplier of retail analytics cannot advise the retailer what should be done to improve the sales, as they are not the domain expert in retail. It’s up to the retailer to utilise the data – for instance, to re-allocate staff in the outlet through the busiest period of the day – this may help the retailer to reduce costs by ensuring there are sufficient staff to keep up with customer demand.”
At Imagus, Fraser Larcombe says he doesn’t believe we’ve reached a tipping point when it comes to the application of intelligent video analytics to surveillance solutions, arguing there is so much more that analytics can do to help solve customers’ problems in multiple areas of operation.
“The most important features of analytics are those that save time or money for the customer – things like line crossing, loitering or object left/removed, VMD, event searches, face recognition, people searches, retail analytics, license plate recognition – all these can provide a great mix of useful functionality, depending on the use case.
“All our applications are currently centred around facial recognition, as we are finding this the main area of use for the security industry. We are, however, experiencing an extension of the requirements for facial recognition to include recognising VIP’s or persons of interest, or for facial verification rather than recognition. Another area of great interest with our solution is retrospective and historical search capabilities.”
According to Larcombe, some of the pitfalls include camera placement and environmental factors such as lighting, motion and field of view limitations.
“Managing customer expectations is key and where most people fail,” he explains. “Some manufacturers promote industry standard benchmark reports as being important in their facial recognition software performance metrics, however, in doing so they can inadvertently promote unrealistic results when used in real world use cases, especially in the surveillance and security industries.
“These industry benchmarking standards are usually derived from a controlled testing environment utilizing high-quality image data sets, when this is not representative of real world scenarios. It is important to understand that some reports in the industry are more about verification than recognition. In the security and surveillance industry, recognition is key, not verification.”
Larcombe argues that analytics does deliver a return on investment if used correctly.
“The idea of retail analytics is a good example here,” he says. “By knowing things like dwell times, movement direction, gender and age, the etail environment can change to suit. This might be to guide more people to an area that’s quiet, or to charge a premium for retail space that’s more utilized.”
In-camera or in-software – what’s the best option?
“While an in-camera option would be utopia, the graphics chips (GPU)in a camera are not currently capable of working as efficiently as software on a server,” Larcombe explains. “If they were capable, and the cameras could offer shared information, the ability to share pictures on a cross platform basis, and in addition to being brand agnostic, then this would be a great leap forward. However, camera manufacturers working together on broad analytics functionality would be very difficult in my opinion, as the camera manufacturers are skewed to the belief that they have all the parts necessary to create a total solution for every customer.”
Does Larcombe think that in the future, it’s possible surveillance solutions will become virtually automated?
“We would like to think so, but realistically there will always be points where it is best to send some information for clarification to a human.”
For Rob Marsden of integration company Addictive Technologies, if anything is holding intelligent video analytics back, it’s user education.
“There is still a huge education gap and a lot of work to be done to educate audiences on the massive value that intelligent VA systems can provide for their organisation,” Marsden says. “For any educated end user who has had an intelligent video analytics solution installed for them, you will not be able to sell them on any system not surpassing their prior experience.”
A key aspect of any IVA application is that integrators and end users should always test and optimise for accuracy, Marsden says.
“The ability for a system to ‘self-learn’ or be ‘trained’ on any particular IVA is key,” he says. “Beware of the temptation to overpromise or oversell the ability, make sure you know what you’re talking about such as the difference between facial recognition and facial detection.
“Regardless of the IVA functionality – in my opinion all are valuable and should be able to be implemented to work seamlessly together where required – it’s all wasted if the ability to search and use those functions is not well thought out, quick and intuitive for the end user to operate, or if IVA events are not brought to the attention of the operator in a user-friendly manner.
According to Marsden, Addictive focuses on deploying systems with IVA as part of the overall security posture of a solution.
“The ability to perform alerts on people as opposed to random object movements in a given area is paramount to a basic integrated solution for us,” he says. “We already implement IVA-based systems that can escalate and contact various levels of response based on the event type and severity.”
Marsden believes integrating functionality suites such as retail analytics enhances return on investment.
“Yes, it does,” he says. “And that’s why I believe the IVA smarts should run at the edge on the camera. IVA will be increasingly common across multi-site retail chains. If cameras contain the smarts, then the software can be focused on providing the ultimate viewing, searching and reporting operator experience.”
IVA certainly has the capacity to change the way we do video surveillance. How realistic are some of the most recent claims we’ve seen about IVA in the general press? I think it’s fair to argue that some of these capabilities are possible in the lab but it’s still the general functionalities that are going to offer reliable performance in the real world. And to get that performance you’re going to need thoughtful installation of quality hardware in the right places.
At SecTech Camera Shootout in Adelaide (you see it at Perth Crown next Wednesday May 23 at 3pm), Fraser Larcombe of Imagus underlined the importance of camera quality when looking for operational IVA.
“You’re not going to get consistently reliable IVA using low cost cameras with wide angle lenses – you won’t be identifying people from one side of an image to another and throughout a camera’s entire depth of field – that’s not realistic. Instead you need quality cameras carefully installed at choke points to ensure the best possible face recognition performance,” Larcombe said.
That observation has certainly been borne out at SecTech where we’ve seen that properly installed IVA integrated into a capable VMS really does work. The secret is to make sure you play to its growing strengths.
