Real-time data integration to the cloud

Sankar Nagarajan www.cloudshoring.in Refer my related blog post at http://www.cloudshoring.in

Cloud Services/Apps/Smart gridsS Complex Event processing Cloud Database Sensor data processing Alarm Processing Cloud Hadoop Map/R Jobs Analysis Email,SMS,Phone Notifications Cloud HPC Jobs Real-time Data & Cloud Ref. www.commsvr.com OPC-UA data integration Cloud ERP/CRM ,Dashboard Application logic OPC UA - Frictionless Bridge

Medical Devices/Systems Industrial /Plan Automation Building Management Automotive Systems CLOUD COMPUTING SERVICE (IaaS/PaaS) Sensor Info Process data Events Real-time Expert Applications/Smart Grids etc Analytics Communications Web,Email ,SMS,Mobile,Twitter,IM etc Real-time Information/Data Processing/CEP

Process and Manufacturing plants have massive historical data + Continuous stream of sensor data, Process and Alarm events etc. Big Data indexing, mining ,Analysis, Combining Complex Event processing etc.

The basic problem to be addressed is that of analysis. The sheer amount of data/info.that needs to be managed can be very large. There's data explosion . The challenge is no longer collecting the information. It's about how to analyze live data in a holistic manner The highlight is that big data is about volume, the velocity with which the data travels in and out, and the variety or the number of different data types and sources that are being indexed and managed The data may have to be analyzed in real-time to make decisions before it is saved. The ability to react immediately in real time would be needed to provide very early warnings or remediation actions. Caution : By real-time ,I mean near-realtime scenarios as dealing with the characteristics of hard real-time systems is out of scope .

Correlate real-time sensor, plant or alarm data with existing Big data (Historical archives) Analyzing similarities in alarm and fault data. E.g.“Bad Actor” alarm Filtering and resolution (Fast and Smart) Distributed Grep :- Plant data Log stats & analysis Find critical trends of plant or process behavior : provide analytics and recommendations (Improved decision making and time to act) Machine learning :- Plant data information classification, Pattern recognition and predictions (Production or supply chain optimisation,Risk management)

It’s no surprise to that data is growing quickly. An IDC study last year confirmed that data is growing faster than Moore’s Law . This means that however you’re processing data today, tomorrow you’re going to be doing it with many more servers….! Clusters will continue to expand within the IT environments. With massive amounts of Plant and process data streaming in ,It is time for Manufacturing and process Industries to leverage Cloud computing to Optimize their IT infrastructure to deal with this effectively Reduce risks (missed opportunities, revenues and disasters) Accelerate innovation in business Derive higher value and returns

CEP event correlation engines ( event correlators ) analyze a mass of events, pinpoint the most significant ones, and trigger actions Enable better Operational Intelligence (OI) solutions to provide insight into business operations by running query analysis against real-time/live feeds and event data streams. “ Regular events normally represents a concrete state, a complex event is normally an aggregation of multiple events (not necessarily of the same type) that identify a meaningful event.”

Process and analyze location (GPS) & other onboard sensor data from automotive systems against dynamic weather & traffic conditions or routes and provide pro-active nofications and actions (SMS,Voice) if problems were determined. In the event of a vehicle breakdown ,determine and find the location co-ordinates and send information about the nearest vehicle towing service/repair shops,Police stations (SMS,Voice,Map info) to the occupants. In the event of an accident (detected through suitable onboard vehicle sensors and validation),Calculate the location co-ordinates and notify Emergency evacuation, medical services, Police and relatives with fine grained information. (SMS,Voice,Email,Fax) . Vital Physical parameters may also be sent if possible. Process field information/data to optimize medical emergency handling in hospitals… (e.g. ambulance disptach,location tracking, doctor notifcations,preparedness assessment and recommendations etc) Process alarm data from Building management systems and send remote alert notifications. Take remedial control actions through SMS or Voice based responses. Seep through sensor data streams to analyse energy consumption trends and make recommendations for resource optimization

Smart grids dealing with digital consumer and industrial power and energy management typically needs a lot of real-time field sensor data . There is an increasing demand to leverage cloud computing and integrate real time data to implement next generation smart grids. Example Drivers .. Increasingly, enterprise clients are concerned about rising utility expenses but they have little or no visibility into the consumption patterns at the plug level. With plug-loads now representing more than 30% of a commercial building’s energy use, the ThinkEco Enterprise Solution provides micro-level data, analytics and control so that clients can continue to improve their energy-consumption strategies and optimize electronic asset ownership - Thinkeco Inc , http://bit.ly/sNxr8J

Smart utilitity meter data management AMI (advanced metering infrastructure) is likely to grow as per IDC’s forecast Intelligent Home energy management Intelligent building control Real time sensor monitoring and data processing Distribution Generation & Automation Load control & Demand response. Manage and control the energy demands of electric vehicles. Gigaom has published an interesting article today on upcoming Smart grid startups some of which seems to have an alignment of their product or services with real-time sensor data and cloud computing thoughts that I have shared. I suggest reading Gigaom’s article for more information and visiting the website and blogs of the companies cited. ( eMeter,Ecologic Analytics,Opower,Control4,Axeda,First fuel software,Regen energy,GridMobility )

Promising Enabling TECHNOLOGIES & TOOLS, CLOUD SERVICES OPC-UA (Sensor and RT interfacing) : HBSoft ,Unified automation,Matrikon,Iconics,QNX OS,Tenasys,Embedded labs Cloud Services ,Tools Amazon AWS Cloud,EC2 Clustering,EC2 Autoscaling,AWS Import/Export,AWS S3,EBS, AWS direct connect , SQS,EMR,AWS VPC Gigaspaces XAP,Windows Azure,Google App engine (GAP) Private & Hybrid cloud : VMWare,Openstack,Cloud.com,Open Nebula Query and Big data processing : Hbase,Apache Hadoop, Cassendra,Redis. Machine learning and Pattern analysis : Apache Mahout Real time Web I/O : Web sockets,XMPP,Zero MQ,Node.js, Atmosphere CEP :- ESPER,Oracle CEP,OpenPDC,Streambase MOM Infrastructure : Apache Camel, Rabbit MQ,Oracle ESB Web & Mobile :- Web sockets,JS,AJAX,HTML5,Android,Ios,Blackberry Critical enabler. OPC within embedded RTOS and Chips are interesting

Huge computing power and data storage availability No upfront IT investments, No need to pre-invest in IT infrastructure of certain scale (either start small and scale based on growth or dynamically scale on demand) Lower (or optimize) data storage costs Improved utilization and reuse of existing IT infrastructure (rationalization) Rapid development and time to deliver Timely access to information Dynamic Process and Business optimization Improved productivity and efficiencies Improved insights and decision making possibilities Improved risk management (mitigation and reduction) Accelerated innovation, Improved ROI Improved business agility Reduction in carbon footprint

There is a huge opportunity to tap across the eco-system for different types of players.For instance., New markets and opportunities for OPC-UA stack providers : HBSoft,Softing so on.. Private Cloud providers can find new markets in this space.(Citrix,Dell(Openstack),VMWare so on) Opportunities beckon Hadoop stack providers Cloudera , MapR,Hortonworks There will be increasing demand for Hadoop-Cloud services: AWS- EMR ,IBM Infosphere, Azure Potential for ‘CEP Service’ clouds to emerge (CEP PaaS ? ,CEP MSPs?) Increasing demand for Multichannel Cloud communication providers AWS-SNS,Tropo,Twilio

ISVs developing SCADA Software/Tools.(A whole new form of Cloud based SCADA ( sub)systems,Smart grid SaaS and niche mobile services can emerge) Niche online and mobile Service providers (Mashups based on GPS,Automotive systems sensors data,Building management systems data,Multi channel notifications services so on…) Traditional example : www.controlsee.com IT Services and Systems integration companies have significant opportunities to engineer the right solutions to deliver niche real-time cloud solutions Technology consultants and Software developers with skills pertaining to this area .(OPC-UA,.Net,Java,Hadoop,Cloud,Web sockets,Amazon AWS Cloud APIs,Webservices so on…)

OPC Foundation Softing OPC UA Architecture Embedding smart communications into inexpensive field devices ARM & Embedded Labs: Redefining Industrial Automation Systems at EW 2011 MapR: Fast, Big and Focused Machine Learning with MapR Choosing Consistency www.gigaom.com

Thank you [email_address] www.cloudshoring.in

Real-time data integration to the cloud

More Related Content

What's hot

What's hot (20)

Similar to Real-time data integration to the cloud

Similar to Real-time data integration to the cloud (20)

More from Sankar Nagarajan

More from Sankar Nagarajan (18)

Recently uploaded

Recently uploaded (20)

Real-time data integration to the cloud