Skip to main content

Apache Airflow

JAMS can integrate with Apache Airflow® to schedule workflows as Directed Acyclic Graphs (DAGs). You can select a DAG from a list and then monitor its status from JAMS. The Job Log tab shows additional information about the Job execution in JAMS.

note

The default polling/Airflow Status Check Interval is 5 seconds, and it can be adjusted to a minimum of 1 second. Specifying a low interval will increase load on the Apache Airflow server and may impact system performance or stability.

You can use the JAMS Connection Store to hold the connection details and reference them in one or more JAMS Jobs. This allows the connection information to be used across multiple Jobs at one time. Also, you would need to update it in only one place.

note

To use the Airflow Execution Method, ensure the JAMS Scheduler and JAMS Client are running 7.8.3 or higher.

Before You Begin​

Before you start to set up the integration with Airflow, ensure you have the following:

  • The name or IP address of the Apache Airflow server.
  • The Airflow server is configured to use HTTPS to ensure secure communication to the REST API.
  • The x.509 certificate used for HTTPS communication with Apache Airflow. If you are using a self-signed certificate, the certificate should be installed on the JAMS Scheduler and any JAMS Agents if they are used to run the Airflow Jobs. Import the certificate .cer file under Trusted Root Certification Authorities in the Certificate Manager.
  • The V2 of the Airflow REST API is used.
  • The username and password for authenticating through the Airflow REST API.
  • The port your Apache Airflow server is using for the REST API. The default is 8080.

Creating Credentials​

You need two Credentials in JAMS to run an Airflow Job:

  • A Credential is used to store the username and password for authenticating through the Airflow REST API.
  • A Credential is used to run the Job in JAMS.

See Working with User Credential Definitions for more information on creating a Credential.

Creating a Connection Store Item for Apache Airflow​

To connect to Airflow, you need a Connection Store item in JAMS. As part of this configuration, you will also create a Credential in JAMS to store the login information for Apache Airflow. See Connections for Integrations for the steps to configure the Connection.

Adding a Credential for Running the JAMS Job​

This Credential must be a valid user on the Windows machine.

  1. Click Credentials from the Shortcuts menu.
  2. Click +.
  3. In the Credential Name field, enter a name for the Credential.
  4. In the Logon As field, enter the username for the Credential.
  5. In the Enter Password and Re-Enter Password fields, enter the password.
  6. In the Edit credential after adding field, clear the checkbox.
  7. Click Ok.

Creating a Job in JAMS for Apache Airflow​

  1. Click Definitions from the JAMS Shortcuts menu.
  2. In the Folder Navigator, click a Folder where the Job will be saved. You may need to create a Folder if one does not exist. See Working with Folders.
  3. Click +.
  4. In the Name field, enter a name for the Job.
  5. In the Execution Method field, select Airflow.
  6. Click Ok.
  7. In the Airflow Connection field, select the Connection Store item that you created for Airflow.
  8. In the DAG Name field, select or enter the DAG that you want to run.
  9. Click the Schedule tab to add schedule items to set when the Job will run.
  10. Click the Properties tab.
  11. In the Execute As field, select the JAMS Credential to run the Job.
  12. In the Agent field, select the Agent if the Job will not run on the Scheduler.
  13. Click Save and Close.

Canceling the Job​

You can cancel a JAMS Job that is running an Airflow DAG from JAMS.

  1. Click Monitor from the JAMS Shortcuts menu.
  2. Right-click the Job that is running and select Cancel.
  3. Click Yes.