distributed - Airflow Scheduler not picking up DAG Runs

I'm setting up airflow such that webserver runs on one machine and scheduler runs on another. Both share the same MySQL metastore database. Both instances come up without any errors in the logs but the scheduler is not picking up any DAG Runs that are created by manually triggering the DAGs via the Web UI. The dag_run table in MysQL shows few entries, all in running state:mysql> select * from dag_run;+----+--------------------------------+----------------------------+---------+------------------------------------+------------------+---------...Read more

rx java - rxjava: queue scheduler with default idle job

I have a client server application and I'm using rxjava to do server requests from the client. The client should only do one request at a time so I intent to use a thread queue scheduler similar to the trampoline scheduler.Now I try to implement a mechanism to watch changes on the server. Therefore I send a long living request that blocks until the server has some changes and sends back the result (long pull).This long pull request should only run when the job queue is idle. I'm looking for a way to automatically stop the watch request when a r...Read more

scheduler - schedulercontext quartz.net

I have several jobs scheduled with Quartz.net. I have the same job detail being repeated for every job like the email address, user id....Each job may have a different trigger so I cannot really put the job detail in each trigger. Is there some other place like the schedulingcontext where I can put the common information once instead of repeating for each job?If yes, can someone please provide a sample of the xml that will be used by quartz.Pratik...Read more

scheduler - Apache airflow: setting catchup to False is not working

I have a DAG created on Apache airflow. It seems scheduler is configured to run it from June on 2015 (By the way. I do not know why, but it is a new DAG y created and I didn't backfill it, I only backfilled other dag with different DAG ID with these date intervals, and the scheduler took those dates and bakfilled my new dag. I'm starting to work with airflow).(Update: I realized DAG is backfilled because start date is set on the DAG default config, although this does not explain the behaviour I expose below)I'm trying to stop the scheduler to r...Read more

Web2py Scheduler Waiting for a task to complete before starting a new task launched from different UI

I have a small web application and it process background task. The task takes 5 minutes to complete. So i have used scheduler.I have configured scheduler as service in a ubuntu machine.[Unit]Description=Web2Py scheduler service[Service]ExecStart=/usr/bin/python /home/www-data/web2py/web2py.py -K myappType=simple[Install]WantedBy=multi-user.targetNow this is a webapp and the long running process is launched from UI. Now when this process is launched from multiple browser, it seems the scheduler waits for the tasks sequentially to be completed in...Read more

scheduler - How to flush out jobs on a Mesos slave?

I want to take out a host (mesos-slave) from the mesos cluster in a clean manner by draining out the executors its running. Is it possible for mesos-master to not give any further work to a mesos-slave but still receive updates for the currently running executors? If thats possible, I can make mesos-master not give anymore work to this slave and once the slave is done with its currently running executors, I can take it out. Please feel free to suggest a better way of achieving the same thing....Read more

Airflow scheduler keep on Failing jobs without heartbeat

I'm new to airflow and i tried to manually trigger a job through UI. When I did that, the scheduler keep on logging that it is Failing jobs without heartbeat as follows:[2018-05-28 12:13:48,248] {jobs.py:1662} INFO - Heartbeating the executor[2018-05-28 12:13:48,250] {jobs.py:1672} INFO - Heartbeating the scheduler[2018-05-28 12:13:48,259] {jobs.py:368} INFO - Started process (PID=58141) to work on /Users/gkumar6/airflow/dags/tutorial.py[2018-05-28 12:13:48,264] {jobs.py:1742} INFO - Processing file /Users/gkumar6/airflow/dags/tutorial.py for t...Read more

scheduler - Trigger Control-M folder but it should ignore few jobs in the Folder

I am modifying an existing Control M folder. In that folder, there are 3 jobs in sequence - (Job1 -> Job2 -> Job3). I want to disable 1st and 3rd job so that those do not get triggered(or at least does not execute anything) when Folder is triggered. I do not want to permanently delete the job.Is there a way to leave a job in Folder and Control M will ignore/bypass that job when Folder is triggered?...Read more

scheduler - Why airflow dag schedule_interval */1 * * * * failed?

I'm setting up docker airflow from apache/airflow(2.0.0 dev). dag code like thisDAG_NAME='Simple_Dag_1'default_args = { 'owner': 'airflow', 'depends_on_past': True, 'start_date': utils.dates.days_ago(2)}dag = DAG(DAG_NAME, schedule_interval='*/1 * * * *', default_args=default_args)run_this = BashOperator( task_id='run_this_first', bash_command='date >> /tmp/airflow.log', dag=dag,)I think it's should be output by minute22:26:0022:27:0022:28:00but it' output like:22:26:4722:27:0122:27:1322:27:2222:27:3922:27:5422:28:1022:2...Read more

scheduler - How to check if a Blue Prism Server service is running?

I am trying to use the scheduler in Blue Prism and followed this guide to configure everything. However, my scheduled tasks are not running (nothing is started, there is no recent activity). One of the requirements to run a schedule is to have the blue prism server service running. How do I check this? I have attached here some screenshots of my configured scheduler. The pending status on the timetable just disappears when I do "Run (schedule) now", and a checkmark never appears. The logs are also empty with no start or end time....Read more