Skip to main content

Apache Airflow

JAMS can integrate with Apache Airflow® to schedule workflows as Directed Acyclic Graphs (DAGs). You can select a DAG from a list and then monitor its status from JAMS. The Job Log tab shows additional information about the Job execution in JAMS.

note

The default polling/Airflow Status Check Interval is 5 seconds, and it can be adjusted to a minimum of 1 second. Specifying a low interval will increase load on the Apache Airflow server and may impact system performance or stability.

You can use the JAMS Connection Store to hold the connection details and reference them in one or more JAMS Jobs. This allows the connection information to be used across multiple Jobs at one time. Also, you would need to update it in only one place.

note

To use the Airflow Execution Method, ensure the JAMS Scheduler and JAMS Client are running 7.8.3 or higher.

Before You Begin​

Before you start to set up the integration with Airflow, ensure you have the following:

  • The name or IP address of the Apache Airflow server.
  • The Airflow server is configured to use HTTPS to ensure secure communication to the REST API.
  • The x.509 certificate used for HTTPS communication with Apache Airflow. If you are using a self-signed certificate, the certificate should be installed on the JAMS Scheduler and any JAMS Agents if they are used to run the Airflow Jobs. Import the certificate .cer file under Trusted Root Certification Authorities in the Certificate Manager.
  • The V2 of the Airflow REST API is used.
  • The username and password for authenticating through the Airflow REST API.
  • The port your Apache Airflow server is using for the REST API. The default is 8080.

Creating a Connection Store Item for Apache Airflow​

To connect to Airflow, you need a Connection Store item in JAMS. As part of this configuration, you will also create a Credential in JAMS to store the login information for Apache Airflow. See Connections for Integrations for the steps to configure the Connection.

Adding a Credential for Running the JAMS Job​

This Credential must be a valid user on the Windows machine.

  1. Expand Execution from the main menu.
  2. Click Credentials.
  3. Click Add.
  4. In the Display Name field, enter a name for the Credential.
  5. In the Logon Username field, enter the username for the Credential.
  6. In the Password and Reenter Password fields, enter the password.
  7. Click Save and Return to List.

Creating a Job in JAMS for Apache Airflow​

  1. Click Jobs from the main menu.
  2. Click a Folder in the Folder Navigator where the Job will be saved.
  3. Click Add.
  4. In the Job Name field, enter a name for the Job.
  5. In the Execution Method field, select Airflow.
  6. Click Save and Continue to Edit.
  7. Click the Source tab.
  8. In the Airflow Connection field, select the Connection Store item that you created for Airflow.
  9. In the DAG Name field, select or enter the DAG that you want to run.
  10. Click the Schedule tab to add schedule items to set when the Job will run.
  11. Click the Properties tab.
  12. In the Execute As field, select the JAMS Credential to run the Job.
  13. In the Agent field, select the Agent if the Job will not run on the Scheduler.
  14. Click Save.

Canceling the Job​

You can cancel a JAMS Job that is running an Airflow DAG from JAMS.

  1. Click Monitor from the main menu.
  2. Click the Job that is running and select Cancel.
  3. Click Yes.