Cloud Native Data Pipelines

Sid Anand | GOTO Chicago 2017


Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, network topology, and compute and storage placement, they need to closely manage their core technology. In recent years, many companies with Big Data needs have started migrating to one of the public cloud vendors. How does the public cloud change the game? Specifically, how can companies effectively marry cloud best-practices with big data technology in order to leverage the benefits of both? Agari, a leading email security company, is applying big data best practices to both the security industry and to the cloud in order to secure the world against email-bourne threats. We do this by building both batch and stream processing predictive data pipelines in the AWS cloud. Come to this talk to learn about our architectural best practices and technologies.

About the speakers

Sid Anand
Sid Anand

Data Architect for Datazoom, a rising email security company. Committer, Advisor, Speaker. Former LinkedIn, Netflix and eBay

Related topics