Apache Beam
A unified model to define and execute data processing pipelines
Apache Beam provides an advanced unified programming model, allowing you to implement batch and streaming data processing jobs that can run on any execution engine. It is easy to use with Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow among other distributed processing back-ends.
Apache Beam developed out of a number of internal Google technologies, including MapReduce, FlumeJava, and Millwheel. Google donated the code to the Apache Software Foundation in 2016, and Googlers continue to contribute regularly to the project.
Apache, Apache Beam and the orange letter B logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
How Google uses Apache Beam
Apache Beam is an open source successor of the SDKs used internally to power the majority of Google’s massive-scale data processing pipelines.
-
Repo
-
License
-
Categories
-
Languages
Related Projects
Firebase SDK
Firebase is an app development platform that provides integrated tools to help you build, grow and monetize your apps. The Firebase SDK enables access to the Firebase services in an intuitive and idiomatic manner on several platforms.
Kubernetes
Kubernetes is a cluster management system for managing containerized applications across multiple hosts, providing mechanisms for deployment, maintenance, and scaling of applications. It groups containers that make up an application into logical units for easy management and discovery.
Knative
Knative components build on top of Kubernetes, abstracting away the complex details and enabling developers to focus on what matters. Built by codifying the best practices shared by successful real-world implementations, Knative solves the “boring but difficult” parts of deploying and managing cloud native services so you don’t have to.
Istio
Istio is an open platform that provides a uniform way to connect, manage, and secure microservices. Istio supports managing traffic flows between microservices, enforcing access policies, and aggregating telemetry data, all without requiring changes to microservice code.