Deploying Airbyte
The Airbyte platform is a sophisticated data integration platform that enables you to handle large amounts of data movement. To quickly deploy Airbyte on your local machine you can visit the Quickstart guide. If setting up an Airbyte server does not fit your use case needs (i.e. you're using Jupyter Notebooks or iterating on an early prototype for your project) you may find the PyAirbyte documentation useful.
Enterprise Customers should follow the steps outlined in our docs on Airbyte Self-Managed Enterprise and the associated implementation guide.
Understanding the Airbyte Deployment
Airbyte is built to be deployed into a Kubernetes cluster. You can use a Cloud Provider, such as, AWS, GCP, Azure, or onto a single node, such as an EC2 VM, or even locally on your computer.
We highly recommend deploying Airbyte using Helm and the documented Helm chart values.
Helm is a Kubernetes package manager for automating deployment and management of complex applications with microservices on Kubernetes. Refer to our Helm Chart Usage Guide for more information about how to get started.
The Infrastructure section describes the Airbyte's recommended cloud infrastructure to set up for each supported platform. Keep in mind that these guides are meant to assist you, but you are not required to follow them. Airbyte is designed to be as flexible as possible in order to fit into your existing infrastructure.
Installation Guide
This installation guide walks through how to deploy Airbyte into any kubernetes cluster. It will run through how to deploy a default version of Airbyte. It will, as an optional step, describe you how you can customize that deployment for your cloud provider and integrations (e.g. ingresses, external databases, external loggers, etc).
This guide assumes that you already have a running kubernetes cluster. If you're trying out Airbyte on your local machine, we recommend using Docker Desktop and enabling the kubernetes extension. We've also tested it with kind, k3s, and colima. If you are installing onto a single vm in a cloud provider (e.g. EC2 or GCE), make sure you've installed kubernetes on that machine. This guide also works for multi-node setups (e.g. EKS and GKE).
1. Add the Helm Repository
The deployment will use a Helm chart which is a package for Kubernetes applications, acting like a blueprint or template that defines the resources needed to deploy an application on a Kubernetes cluster. Charts are stored in helm-repo.
To add a remote helm repo:
-
Run:
helm repo add airbyte https://airbytehq.github.io/helm-charts. In this example,airbyteis being used to represent the name of the repository that will be indexed locally. -
After adding the repo, perform the repo indexing process by running
helm repo update. -
You can now browse all charts uploaded to repository by running
helm search repo airbyte
An example of the chart output:
NAME CHART VERSION APP VERSION DESCRIPTION
airbyte/airbyte 0.290.0 0.63.6 Helm chart to deploy airbyte
airbyte/airbyte-api-server 0.290.0 0.63.6 Helm chart to deploy airbyte-api-server
airbyte/airbyte-bootloader 0.290.0 0.63.6 Helm chart to deploy airbyte-bootloader
airbyte/airbyte-cron 0.40.37 0.40.17 Helm chart to deploy airbyte-cron
airbyte/airbyte-workload-api-server 0.49.18 0.50.33 Helm chart to deploy airbyte-api-server
airbyte/connector-builder-server 0.290.0 0.63.6 Helm chart to deploy airbyte-connector-builder-...
airbyte/cron 0.290.0 0.63.6 Helm chart to deploy airbyte-cron
airbyte/keycloak 0.290.0 0.63.6 Helm chart to deploy airbyte-keycloak
airbyte/keycloak-setup 0.290.0 0.63.6 Helm chart to deploy airbyte-keycloak-setup
airbyte/metrics 0.290.0 0.63.6 Helm chart to deploy airbyte-metrics
airbyte/pod-sweeper 0.290.0 0.63.6 Helm chart to deploy airbyte-pod-sweeper
airbyte/server 0.290.0 0.63.6 Helm chart to deploy airbyte-server
airbyte/temporal 0.290.0 0.63.6 Helm chart to deploy airbyte-temporal
airbyte/webapp 0.290.0 0.63.6 Helm chart to deploy airbyte-webapp
airbyte/worker 0.290.0 0.63.6 Helm chart to deploy airbyte-worker
airbyte/workload-api 0.50.3 0.50.35 Helm chart to deploy the workload-api service
airbyte/workload-api-server 0.290.0 0.63.6 Helm chart to deploy the workload-api service
airbyte/workload-launcher 0.290.0 0.63.6 Helm chart to deploy airbyte-workload-launcher
2. Create a Namespace for Airbyte
While it is not strictly necessary to isolate the Airbyte installation into its own namespace, it is good practice and recommended as a part of the installation.
This documentation assumes that you chose the name airbyte for the namespace, but you may choose a different name if required.
To create a namespace run the following:
kubectl create namespace airbyte
3. Create a values.yaml override file
To configure your installation of Airbyte, you will need to override specific parts of the Helm Chart. To do this you should create a new file called values.yaml somewhere that is accessible during the installation process.
The documentation has been created to "build up" a values.yaml, so there is no need to copy the whole of the Chart values.yaml. You only need to provide the specific overrides.
Each Integration will provide a section of the specific values that you should override and provide examples of what the values should look like. An example values.yaml file may look like the following:
global:
airbyteUrl: https://airbyte.company.example
4. (optional for customized installations only) Customize your deployment
The Airbyte platform is built to integrate with your existing cloud infrastructure. You can configure various components of the platform to suit your needs. This includes an object stores, such as S3 or GCS for storing logs and state, a database for externalizing state, and a secrets manager for keep your secrets secure.
Each of these integrations will require you to create a secret upfront. For instructions on how to do that check out Creating a Secret
Each of these integrations can be configured to suit your specific needs and is described in the Integration section. Each of these integrations has its own section where you'll find an explanation for why it's useful to configure the integration. There, you'll also find details about how to configure the integration.
Before you can configure this stuff in a cloud provider, you need to set up your policies:
After your policies are set up, here's a list of customizations.
5. Installing Airbyte
After you have applied your Secret values to the Cluster and you have filled out a values.yaml file appropriately for your specific configuration, you can begin a Helm Install. To do this, make sure that you have the Helm Client installed and on your path. Then you can run:
helm install \
airbyte \
airbyte/airbyte \
--namespace airbyte \
--values ./values.yaml
After the installation has completed, you can configure your Ingress by following the directions for your specific Ingress provider.
6. Set up port forward for UI access
Helm install with spit out instructions for how to set up the port forward. Go ahead and run that command. It should look something like this:
export POD_NAME=$(kubectl get pods --namespace airbyte -l "app.kubernetes.io/name=webapp" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace airbyte $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace airbyte port-forward $POD_NAME 8080:$CONTAINER_PORT
You can now access the UI in your browser at: http://127.0.0.1:8080.
If you'd like to set up a more permanent ingress checkout our ingress customization. For a deployment to a local machine we recommend using nginx as an easy-to-setup option.
As part of maintainging your Airbyte instance, you'll need to do periodic upgrades. See our documentation on when and how to upgrade Airbyte for details.