HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA

•

0 likes•269 views

Keyur has worked with data for over 15 years in a number of consultancy, technical and leadership roles. He’s passionate about Cloud and Big Data and has a proven track record of architecting and delivering business “step changes” using the two. He currently heads up Data Architecture function at Betfair driving the company’s journey to Big Data Cloud.

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
HOW TO LEVERAGE CLOUD
FOR
BIG DATA ANALYTICS
By Keyur Mehta
Head of Data Architecture, Betfair
Big Data Week Conference, London, Nov 2015

Betfair Story and Data Challenges
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

Betfair Story and Data Challenges
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
100M
Transactions
Per Day
99.9%
transactions
within a sec
£55 Billion
Trading
value
Billions of
Weblog
Events
Per Day
One of
Europe’s
largest
.com IPO
Start up
to
FTSE 250
Start up to one
of Europe’s
biggest .com
IPO

On Premise
Traditional
Data Warehouse
Offloading Analytics
to
Active Big Data Lake
Leveraging
Cloud to support
Business Processes
Betfair’s journey to the Big Data cloud
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

Big Data in the Cloud
What do we use it?
STORAG
E
MPP
HADOOP
VISUALIZATION
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

On demand Hadoop cluster
Spin up and shut down
Choice of instance types
Spot instances
Ephemeral - EMRFS vs HDFS
Store once, Process in multiple Clusters
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Big Data in the Cloud
AWS EMR (Hadoop)
Speed
Elasticity
Transparency
Simplicity

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Big Data Visualization in the Cloud
Tableau

On Premise
Traditional
Data Warehouse
Offloading Analytics
to
AWS Cloud
Leveraging
AWS Cloud to support
Business Processes
Big Data in the Cloud
Betfair Journey – what’s happening?
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Machine
Learning
Social
Streaming
No Technology is Perfect !!!

KEYUR MEHTA
13
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Comments, Feedback, Suggestions or
Want to work with us?
https://uk.linkedin.com/in/keyurmehtadata
keyur.mehta@betfair.com

Big Data in the Cloud
Why Public Cloud? Why AWS?
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange
Limited.
10x bigger than next competitors
combined

Viewers also liked

Advanced data modeling with apache cassandra

Patrick McFadin

Betfair + Couchbase

bloodredsun

- Betfair chose Couchbase as their strategic document NoSQL solution to store session state, cross-session state, and service persistence for key-based entities. Couchbase was selected due to its strong performance, scalability, schema flexibility, and ease of use for both developers and operations teams. Betfair has been using Couchbase in production for about 6 months across multiple application clusters.

betfair-uesummit-may2012-chetana

Chetana Deorah

The document discusses the differences between consistency and coherency in user experience design. It focuses on the challenges of designing for touch-based experiences across multiple platforms and countries. While consistency is important for building brand loyalty, the author argues that user experience design for touch prioritizes problem solving and being context sensitive over strict consistency. The author shares their process for collaborative design that captures an essence while allowing for flexibility across platforms.

Spy v Spy - Treachery in the Dev/Ops Trenches

bloodredsun

This document discusses challenges faced by a company in building a new website to handle high traffic volumes. Some key issues included slow performance of the original Java-based site, difficulties monitoring site performance at scale, outages caused by database failures, and challenges integrating new NoSQL databases and microservices. The company addressed these problems through techniques like optimizing database queries, distributed monitoring, database sharding, and testing changes in production. The overall message is that infrastructure must evolve to keep up with rapidly changing code, and resilience is critical when handling large volumes of traffic.

Time Series Data: OpenTSDB and TSP (Betfair)

Rakh1

Netflix Exemplar

phatmcat

The document discusses the information architecture of Netflix, including its organization, navigation, labeling, and search features. It notes that Netflix uses a hybrid genre-based organization scheme with over 20 genres. The structure is narrow and deep, with two views of genres and sub-genres. Navigation is mostly through global and local links within genres. Search is powerful for finding known items but browsing and recommendations are better for exploratory seeking.

openstackreferencearchitecturewhitepaper

Richard Haigh

This document summarizes Paddy Power Betfair's private OpenStack cloud called i2. Key points: 1) i2 provides an extensible, scalable platform to host all development, test, and production applications on OpenStack. It uses software defined networking and automation to improve provisioning speed and flexibility. 2) Major technologies used include Red Hat OpenStack Platform, Arista switching, Nuage Networks SDN, Citrix load balancing, Pure Storage, and HP servers. 3) Benefits include faster provisioning, standard tooling, security, availability, scalability, and improved performance. 4) The approach included vendor selection, proof of concept, a production pilot, and migration of applications to

An Introduction to Priam

Jason Brown

OpenTSDB 2.0

HBaseCon

Speakers: Chris Larsen (Limelight Networks) and Benoit Sigoure (Arista Networks) The OpenTSDB community continues to grow and with users looking to store massive amounts of time-series data in a scalable manner. In this talk, we will discuss a number of use cases and best practices around naming schemas and HBase configuration. We will also review OpenTSDB 2.0's new features, including the HTTP API, plugins, annotations, millisecond support, and metadata, as well as what's next in the roadmap.

Creating a Next-Generation Big Data Architecture

Perficient, Inc.

If you’ve spent time investigating Big Data, you quickly realize that the issues surrounding Big Data are often complex to analyze and solve. The sheer volume, velocity and variety changes the way we think about data – including how enterprises approach data architecture. Significant reduction in costs for processing, managing, and storing data, combined with the need for business agility and analytics, requires CIOs and enterprise architects to rethink their enterprise data architecture and develop a next-generation approach to solve the complexities of Big Data. Creating the data architecture while integrating Big Data into the heart of the enterprise data architecture is a challenge. This webinar covered: -Why Big Data capabilities must be strategically integrated into an enterprise’s data architecture -How a next-generation architecture can be conceptualized -The key components to a robust next generation architecture -How to incrementally transition to a next generation data architecture

Data Stream Processing with Apache Flink

Fabian Hueske

Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts

Acunu

Cassandra By Example: Data Modelling with CQL3

Eric Evans

CQL is the query language for Apache Cassandra that provides an SQL-like interface. The document discusses the evolution from the older Thrift RPC interface to CQL and provides examples of modeling tweet data in Cassandra using tables like users, tweets, following, followers, userline, and timeline. It also covers techniques like denormalization, materialized views, and batch loading of related data to optimize for common queries.

AWS Re:Invent - High Availability Architecture at Netflix

Adrian Cockcroft

Netflix Cloud Architecture and Open Source

aspyker

Overview of Apache Flink: Next-Gen Big Data Analytics Framework

Slim Baltagi

These are the slides of my talk on June 30, 2015 at the first event of the Chicago Apache Flink meetup. Although most of the current buzz is about Apache Spark, the talk shows how Apache Flink offers the only hybrid open source (Real-Time Streaming + Batch) distributed data processing engine supporting many use cases: Real-Time stream processing, machine learning at scale, graph analytics and batch processing. In these slides, you will find answers to the following questions: What is Apache Flink stack and how it fits into the Big Data ecosystem? How Apache Flink integrates with Apache Hadoop and other open source tools for data input and output as well as deployment? What is the architecture of Apache Flink? What are the different execution modes of Apache Flink? Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark? Who is using Apache Flink? Where to learn more about Apache Flink?

Apache Flink: Real-World Use Cases for Streaming Analytics

Slim Baltagi

This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications. In this talk, you learn more about: 1. What is Apache Flink Stack? 2. Batch vs. Streaming Analytics 3. Key Differentiators of Apache Flink for Streaming Analytics 4. Real-World Use Cases with Flink for Streaming Analytics 5. Who is using Flink? 6. Where do you go from here?

Big Data Architecture

Guido Schmutz

This document discusses different architectures for big data systems, including traditional, streaming, lambda, kappa, and unified architectures. The traditional architecture focuses on batch processing stored data using Hadoop. Streaming architectures enable low-latency analysis of real-time data streams. Lambda architecture combines batch and streaming for flexibility. Kappa architecture avoids duplicating processing logic. Finally, a unified architecture trains models on batch data and applies them to real-time streams. Choosing the right architecture depends on use cases and available components.

Viewers also liked (18)

Advanced data modeling with apache cassandra

Betfair + Couchbase

betfair-uesummit-may2012-chetana

Spy v Spy - Treachery in the Dev/Ops Trenches

Time Series Data: OpenTSDB and TSP (Betfair)

Netflix Exemplar

openstackreferencearchitecturewhitepaper

An Introduction to Priam

OpenTSDB 2.0

Creating a Next-Generation Big Data Architecture

Data Stream Processing with Apache Flink

Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts

Cassandra By Example: Data Modelling with CQL3

AWS Re:Invent - High Availability Architecture at Netflix

Netflix Cloud Architecture and Open Source

Overview of Apache Flink: Next-Gen Big Data Analytics Framework

Apache Flink: Real-World Use Cases for Streaming Analytics

Big Data Architecture

Similar to HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA

There are 250 Database products, are you running the right one?

Aerospike, Inc.

This webinar discusses choosing the right database for organizations. It will cover industry trends driving data and database evolution, real-world use cases where speed and scale are important, and an architecture overview. Speakers from Forrester and Aerospike will discuss how new applications are challenging traditional databases and how Aerospike's in-memory database provides extremely high performance for large-scale, data-intensive workloads. The agenda includes an industry overview, tips for choosing a database, how data has evolved, examples where low latency is critical, and a question and answer session.

Get the Most Out of Amazon EC2: A Deep Dive on Reserved, On-Demand, and Spot ...

Amazon Web Services

With Amazon EC2, you have the flexibility to mix-and-match purchasing models to suit your business needs. By combining pay-as-you-go (On-Demand), reserve ahead of time for discounts (Reserved), and high-discount spare capacity (Spot) purchasing models, you can optimize cost, grow your compute capacity and throughput, and enable new types of cloud computing applications. This presentation will guide you on how to achieve high performance and availability at the lowest TCO. We will explore how to best combine EC2's purchasing models across several common applications with immediately actionable takeaways.

E-Commerce Success is a Balancing Act. Ensure Success with ClustrixDB.

Clustrix

Data Culture Series - Keynote - 3rd Dec

Jonathan Woodward

Big data. Small data. All data. You have access to an ever-expanding volume of data inside the walls of your business and out across the web. The potential in data is endless – from predicting election results to preventing the spread of epidemics. But how can you use it to your advantage to help move your business forward? Drive a Data Culture within your organisation Keynote include Ric Howe & Anthony Saxby

BDPA Cincinnati: 'Big Data - Friend or Foe?'

BDPA Education and Technology Foundation

It seems that everyone is talking about Big Data these days. As the Industrial Internet evolves and continues to feed the Big Data machine, companies are finding it more and more critical to develop strategies for turning data into information and information in intelligence. There’s certainly not a shortage of technologies in the marketplace to start playing with the petabytes of data coming from within and outside of the enterprise. This is the powerpoint presentation used by our guest speaker Barbara (Barb) Kruetzkamp, IT Leader in Data Management at GE Aviation, to discuss approaches and frameworks to enhance business intelligence capabilities by linking industrial and enterprise (internal) data. We also compared traditional vs. transformational IT execution models, and how to put data first. Barb was born and raised in Cincinnati, Ohio. She attended Thomas More College in Kentucky, graduating with a B.A. in Computer Science and Business Administration. She built technical depth leading infrastructure architecture, then as a Chief Enterprise Architect at GE Corporate. Barb returned to GE Aviation in 2014 to lead the master data management initiative. Barb enjoys volunteering with the developmentally disabled, STEM and high school band students. She also likes to cook, jazzercise, and travel abroad. Her 3 kids keep life active and fun.

Aerospike Meetup - Introduction - Ami - 04 March 2020

Aerospike

Get the Most Out of Amazon EC2: A Deep Dive on Reserved, On-Demand, and Spot ...

Amazon Web Services

Take Action: The New Reality of Data-Driven Business

Inside Analysis

The Briefing Room with Dr. Robin Bloor and WebAction Live Webcast on July 23, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=360d371d3a49ad256942f55350aa0a8b The waiting used to be the hardest part, but not anymore. Today’s cutting-edge enterprises can seize opportunities faster than ever, thanks to an array of technologies that enable real-time responsiveness across the spectrum of business processes. Early adopters are solving critical business challenges by enabling the rapid-fire design, development and production of very specific applications. Functionality can range from improved customer engagement to dynamic machine-to-machine interactions. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor, who will tout a new era in data-driven organizations, and why a data flow architecture will soon be critical for industry leaders. He’ll be briefed by Sami Akbay of WebAction, who will showcase his company’s real-time data management platform, which combines all the component parts needed to access, process and leverage data big and small. He’ll explain how this new approach can provide game-changing power to organizations of all types and sizes. Visit InsideAnlaysis.com for more information.

Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...

Amazon Web Services

Amazon Kinesis makes it easy to speed up the time it takes for you to get valuable, real-time insights from your streaming data. In this session, we walk through the most popular applications that customers implement using Amazon Kinesis, including streaming extract-transform-load, continuous metric generation, and responsive analytics. Our customer Autodesk joins us to describe how they created real-time metrics generation and analytics using Amazon Kinesis and Amazon Elasticsearch Service. They walk us through their architecture and the best practices they learned in building and deploying their real-time analytics solution.

Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D

Domonkos Tikk

How to Prepare for 5-Minute Settlement: Everything Utilities Traders Need to ...

Kaitlyn Hurley

Kelley Blue Book and Cloud Computing

David Chou

Amazon EC2

Amazon Web Services

The document provides an overview of Amazon EC2 purchasing options including On-Demand, Reserved, and Spot instances. It discusses using the different options for various workload types and provides examples of how companies from different industries utilize the different purchasing models. The key models are: On-Demand for short term needs, Reserved for steady state workloads to get discounts for long term commitments, and Spot for flexible workloads that can be interrupted. It also emphasizes having a balanced approach across the different options.

AWS RoadShow 2013 Curitiba

Amazon Web Services LATAM

This document discusses a technology event promoting Amazon Web Services (AWS) in Brazil. It includes an agenda for keynote speeches on AWS partnerships and architectures for the 21st century. Example benefits of AWS are also listed such as cost reduction through elasticity and optimization of large corporations. Statistics on AWS services and growth of AWS customers in Latin America are provided. The document encourages writing the history of cloud computing in Brazil together and states that the cloud acts as a fertilizer for startups.

Build a Big Data Warehouse on the Cloud in 30 Minutes

Caserta

IoT: Understanding its potential and what makes it tick! by Mark Torr

Bosnia Agile

클라우드에서의 데이터 웨어하우징 & 비즈니스 인텔리전스

Amazon Web Services Korea

The document discusses challenges with traditional data warehousing and analytics including high upfront costs, difficulty managing infrastructure, and inability to scale easily. It introduces Amazon Web Services (AWS) and Amazon Redshift as a solution, allowing for easy setup of data warehousing and analytics in the cloud at low costs without large upfront investments. AWS services like Amazon Redshift provide flexible, scalable infrastructure that is easier to manage than traditional on-premise systems and enables organizations to more effectively analyze large amounts of data.

How to Say Yes to Self-Service in the Cloud and Become an IT Hero

RightScale

We’ve all seen the trend: your internal customers, developers, and business users alike, want self-service access to IT resources they need. And increasingly they want those resources in the cloud via a single management console. Cloud computing offers them the agility they need to respond to market demands and the ability to scale up while keeping costs low. IT can become a hero by providing a self-service portal where internal customers can provision their own cloud resources while IT can maintain the control and visibility an enterprise requires. The happy result: enterprise agility and enterprise control. During this session, learn how to empower your internal customers to provision the necessary cloud resources when they need them but also ensure that they are well within IT approved guidelines. To help illustrate the effectiveness of this approach, our presenters walk you through real-world examples and the overall impact on their organizations. 1. Create an IT vending machine with consistent and reproducible processes. 2. Manage multiple IT environments through a single-pane-of-glass. 3. Use cost planning and forecasting to fine-tune and understand cloud spend. 4. Discover reporting and auditing tools to ensure compliance.

How to Build Continuous Ingestion for the Internet of Things

Cloudera, Inc.

The Internet of Things is moving into the mainstream and this new world of data-driven products is transforming a vast number of industry sectors and technologies. However, IoT creates a new challenge: how to build and operationalize continual data ingestion from such a wide and ever-changing array of endpoints so that the data arrives consumption-ready and can drive analysis and action within the business. In this webinar, Sean Anderson from Cloudera and Kirit Busu, Director of Product Management at StreamSets, will discuss Hadoop's ecosystem and IoT capabilities and provide advice about common patterns and best practices. Using specific examples, they will demonstrate how to build and run end-to-end IOT data flows using StreamSets and Cloudera infrastructure.

Conflict in the Cloud – Issues & Solutions for Big Data

Halo BI

Similar to HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA (20)

There are 250 Database products, are you running the right one?

Get the Most Out of Amazon EC2: A Deep Dive on Reserved, On-Demand, and Spot ...

E-Commerce Success is a Balancing Act. Ensure Success with ClustrixDB.

Data Culture Series - Keynote - 3rd Dec

BDPA Cincinnati: 'Big Data - Friend or Foe?'

Aerospike Meetup - Introduction - Ami - 04 March 2020

Get the Most Out of Amazon EC2: A Deep Dive on Reserved, On-Demand, and Spot ...

Take Action: The New Reality of Data-Driven Business

Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...

Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D

How to Prepare for 5-Minute Settlement: Everything Utilities Traders Need to ...

Kelley Blue Book and Cloud Computing

Amazon EC2

AWS RoadShow 2013 Curitiba

Build a Big Data Warehouse on the Cloud in 30 Minutes

IoT: Understanding its potential and what makes it tick! by Mark Torr

클라우드에서의 데이터 웨어하우징 & 비즈니스 인텔리전스

How to Say Yes to Self-Service in the Cloud and Become an IT Hero

How to Build Continuous Ingestion for the Internet of Things

Conflict in the Cloud – Issues & Solutions for Big Data

More from Big Data Week

BDW17 London - Edward Kibardin - Mitie PLC - Learning and Topological Data A...

HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA

Similar to HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA (20)

More from Big Data Week

More from Big Data Week (20)

Recently uploaded

Recently uploaded (20)

HOW TO LEVERAGE CLOUD FOR BIG DATA ANALYTICS - KEYUR MEHTA