Airflow Hooks Vs Operators

Utf 8 licensed to the apache software foundation asf under one or more contributor license agreements.

Airflow hooks vs operators. The asf licenses this file to you under the apache license version. According to the airflow documentation they can be added using airflow s plugins mechanism. When the operator invokes the query on the hook object a new connection gets created if it doesn t exist. Airflow pools can be used to limit the execution parallelism on arbitrary sets of tasks.

Software operators and hooks. Source code for airflow operators postgres operator coding. An operator represents a single ideally idempotent task. Hooks are also very useful on their own to use in python scripts airflow airflow operators pythonoperator and in interactive environments like ipython or jupyter notebook.

Pools some systems can get overwhelmed when too many processes hit them at the same time. These integrations allow you to perform various operations within software developed by apache software foundation. With latest airflow version 1 10 sshexecuteoperator is deprecated and new sshoperator has to be used. Module contents class airflow contrib operators ssh operator sshoperator ssh hook none ssh conn id none remote host none command none timeout 10 do xcom push false environment none get pty false args kwargs source.

If anyone is using 1 10 then new import should be from airflow contrib hooks ssh hook import sshhook and from airflow contrib operators ssh operator import sshoperator. Custom hooks and operators are a powerful way to extend airflow to meet your needs. See the notice file distributed with this work for additional information regarding copyright ownership. There is however some confusion on the best way to implement them.

See the operators concepts documentation and the operators api reference for more information. This however overcomplicates the issue and leads to confusion for many people. Airflow models baseoperator sshoperator to execute commands on given remote host using the ssh hook.