Deploying Apache Airflow

Apache Airflow is an open source workflow management platform for data engineering pipelines. You can use it to automate, orchestrate, and monitor workflows and data pipelines. One of Airflow’s greatest features is that you can create and execute workflows with code. When you use workflows that are powered by code, you can version control, collaborate on, and debug your workflows.

Deploying a Quick Deploy App

Akamai Quick Deploy Apps let you easily deploy software on a Compute Instance using Cloud Manager. See Get Started with Quick Deploy Apps for complete steps.

1. Log in to Cloud Manager and select the Quick Deploy Apps link from the left navigation menu. This displays the Linode Create page with the Marketplace tab pre-selected.

2. Under the Select App section, select the app you would like to deploy.

3. Complete the form by following the steps and advice within the Creating a Compute Instance guide. Depending on the Quick Deploy App you selected, there may be additional configuration options available. See the Configuration Options section below for compatible distributions, recommended plans, and any additional configuration options available for this Quick Deploy App.

4. Click the Create Linode button. Once the Compute Instance has been provisioned and has fully powered on, wait for the software installation to complete. If the instance is powered off or restarted before this time, the software installation will likely fail.

To verify that the app has been fully installed, see Get Started with Akamai Quick Deploy Apps > Verify Installation. Once installed, follow the instructions within the Getting Started After Deployment section to access the application and start using it.

Configuration Options

• Supported distributions: Ubuntu 24.04 LTS
• Recommended plan: All plan types and sizes can be used.

Apache Airflow Options

• Email Address (required): The email address to use for generating SSL certificates.

Limited User (Optional)

You can optionally fill out the following fields to automatically create a limited user for your new Compute Instance. This is recommended for most deployments as an additional security measure. This account will be assigned to the sudo group, which provides elevated permission when running commands with the sudo prefix.

• Limited sudo user: Enter your preferred username for the limited user.
• Password for the limited user: Enter a strong password for the new user.
• SSH public key for the limited user: If you wish to login as the limited user through public key authentication (without entering a password), enter your public key here. See Creating an SSH Key Pair and Configuring Public Key Authentication on a Server for instructions on generating a key pair.
• Disable root access over SSH: To block the root user from logging in over SSH, select Yes (recommended). You can still switch to the root user once logged in and you can also log in as root through Lish.

Custom Domain (Optional)

If you wish to automatically configure a custom domain, you first need to configure your domain to use Linode’s name servers. This is typically accomplished directly through your registrar. See Use Linode’s Name Servers with Your Domain. Once that is finished, you can fill out the following fields for the Quick Deploy App:

• Linode API Token: If you wish to use the Linode’s DNS Manager to manage DNS records for your custom domain, create a Linode API Personal Access Token on your account with Read/Write access to Domains. If this is provided along with the subdomain and domain fields (outlined below), the installation attempts to create DNS records via the Linode API. See Get an API Access Token. If you do not provide this field, you need to manually configure your DNS records through your DNS provider and point them to the IP address of the new instance.
• Subdomain: The subdomain you wish to use, such as www for www.example.com.
• Domain: The domain name you wish to use, such as example.com.

Obtain the Credentials

Once the app is deployed, you need to obtain the credentials from the server. To obtain the credentials:

1. Log in to your new Compute Instance using one of the methods below:

   • Lish Console: Log in to Cloud Manager, click the Linodes link in the left menu, and select the Compute Instance you just deployed. Click Launch LISH Console. Log in as the root user. To learn more, see Using the Lish Console.
   • SSH: Log in to your Compute Instance over SSH using the root user. To learn how, see Connecting to a Remote Server Over SSH.

2. Run the following command to access the contents of the credentials file:

   cat /home/$USERNAME/.credentials

This returns the admin password and other details that were automatically generated when the instance was deployed. Save them securely. Once saved, you can safely delete the file.

Getting Started After Deployment

Once you’ve obtained the credentials, you can access your Apache Airflow instance and open a browser and navigate to your Linode domain entered during deployment or the rDNS domain https://203-0-113-0.ip.linodeusercontent.com.

Within the Airflow login prompt that appears, enter the credentials provided in the credentials file and sign in.

![Screenshot of the Airflow login prompt](airflow-login.png)

Once you are signed in, the Airflow dashboard appears. From here, you can view the DAGs (Directed Acyclic Graphs) and access all other areas of the dashboard.

![Screenshot of Apache Airflow GUI](airflow-gui.png)

You can now start using Apache Airflow. If you are unfamiliar with it, consider reading through the official documentation or Linode’s own guides:

- [Airflow > Tutorials](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/index.html)
- [Airflow > How-to Guides](https://airflow.apache.org/docs/apache-airflow/stable/howto/index.html)
- [Create Connections and Variables in Apache Airflow](/docs/guides/apache-airflow-tutorial-creating-connections-and-variables/)

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Apache Airflow