Quartz 实现分布式任务调度


声明:本文转载自https://my.oschina.net/tigerlene/blog/1580082,转载目的在于传递更多信息,仅供学习交流之用。如有侵权行为,请联系我,我会及时删除。

情景

在开发的时候可能需要用到任务调度,通常我们使用quartz,单独部署一个实例来进行任务调度,然后就会存在单点的风险,quartz帮我们实现了多台实例时候的高可用,保证同一时间只有一个任务会被执行,下面我们来看一下具体是怎么实现的。

输入图片说明

实现原理

>Quartz’s clustering features bring both high availability and scalability to your scheduler via fail-over and load balancing functionality.

Quartz 实现高可用是通过依赖JDBC-Jobstore (JobStoreTX or JobStoreCMT),来实现的,所以使用这个的前提是每个节点共享同一个数据库()

当任务触发时间开始时,每个实例上的任务会并发获取同一把锁(在后面的代码分析中我们可以看到实际上是通过数据库表的全局锁实现的),谁先获得锁谁就先执行,其它节点的任务就不会执行任务。

如果一个节点挂掉了,而且任务被标记为recovery,就会被其它节点检测到重新执行这个任务,没有被标记的话,就会再下一次任务触发的时候重新由其它节点执行。

适用情况

>The clustering feature works best for scaling out long-running and/or cpu-intensive jobs (distributing the work-load over multiple nodes). If you need to scale out to support thousands of short-running (e.g 1 second) jobs, consider partitioning the set of jobs by using multiple distinct schedulers (including multiple clustered schedulers for HA). The scheduler makes use of a cluster-wide lock, a pattern that degrades performance as you add more nodes (when going beyond about three nodes - depending upon your database’s capabilities, etc.).

因为是通过加全局锁来实现的,所以比较适合 任务执行时间较长的定时任务。

使用过程

限制条件

  1. 集群之间的机器要保持时钟同步(大部分分布式系统都有这个要求,不必担心)
  2. 集群之间部分配置文件quartz.properties要保持一致,下面配置会详解

执行指定的数据库脚本

解压jar包之后,在docs/dbTables目录下,根据自己的系统适用不同的脚本,下面以mysql为例

# # Quartz seems to work best with the driver mm.mysql-2.0.7-bin.jar # # PLEASE consider using mysql with innodb tables to avoid locking issues # # In your Quartz properties file, you'll need to set  # org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate #  DROP TABLE IF EXISTS QRTZ_FIRED_TRIGGERS; DROP TABLE IF EXISTS QRTZ_PAUSED_TRIGGER_GRPS; DROP TABLE IF EXISTS QRTZ_SCHEDULER_STATE; DROP TABLE IF EXISTS QRTZ_LOCKS; DROP TABLE IF EXISTS QRTZ_SIMPLE_TRIGGERS; DROP TABLE IF EXISTS QRTZ_SIMPROP_TRIGGERS; DROP TABLE IF EXISTS QRTZ_CRON_TRIGGERS; DROP TABLE IF EXISTS QRTZ_BLOB_TRIGGERS; DROP TABLE IF EXISTS QRTZ_TRIGGERS; DROP TABLE IF EXISTS QRTZ_JOB_DETAILS; DROP TABLE IF EXISTS QRTZ_CALENDARS;   CREATE TABLE QRTZ_JOB_DETAILS   (     SCHED_NAME VARCHAR(120) NOT NULL,     JOB_NAME  VARCHAR(200) NOT NULL,     JOB_GROUP VARCHAR(200) NOT NULL,     DESCRIPTION VARCHAR(250) NULL,     JOB_CLASS_NAME   VARCHAR(250) NOT NULL,     IS_DURABLE VARCHAR(1) NOT NULL,     IS_NONCONCURRENT VARCHAR(1) NOT NULL,     IS_UPDATE_DATA VARCHAR(1) NOT NULL,     REQUESTS_RECOVERY VARCHAR(1) NOT NULL,     JOB_DATA BLOB NULL,     PRIMARY KEY (SCHED_NAME,JOB_NAME,JOB_GROUP) );  CREATE TABLE QRTZ_TRIGGERS   (     SCHED_NAME VARCHAR(120) NOT NULL,     TRIGGER_NAME VARCHAR(200) NOT NULL,     TRIGGER_GROUP VARCHAR(200) NOT NULL,     JOB_NAME  VARCHAR(200) NOT NULL,     JOB_GROUP VARCHAR(200) NOT NULL,     DESCRIPTION VARCHAR(250) NULL,     NEXT_FIRE_TIME BIGINT(13) NULL,     PREV_FIRE_TIME BIGINT(13) NULL,     PRIORITY INTEGER NULL,     TRIGGER_STATE VARCHAR(16) NOT NULL,     TRIGGER_TYPE VARCHAR(8) NOT NULL,     START_TIME BIGINT(13) NOT NULL,     END_TIME BIGINT(13) NULL,     CALENDAR_NAME VARCHAR(200) NULL,     MISFIRE_INSTR SMALLINT(2) NULL,     JOB_DATA BLOB NULL,     PRIMARY KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP),     FOREIGN KEY (SCHED_NAME,JOB_NAME,JOB_GROUP)         REFERENCES QRTZ_JOB_DETAILS(SCHED_NAME,JOB_NAME,JOB_GROUP) );  CREATE TABLE QRTZ_SIMPLE_TRIGGERS   (     SCHED_NAME VARCHAR(120) NOT NULL,     TRIGGER_NAME VARCHAR(200) NOT NULL,     TRIGGER_GROUP VARCHAR(200) NOT NULL,     REPEAT_COUNT BIGINT(7) NOT NULL,     REPEAT_INTERVAL BIGINT(12) NOT NULL,     TIMES_TRIGGERED BIGINT(10) NOT NULL,     PRIMARY KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP),     FOREIGN KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP)         REFERENCES QRTZ_TRIGGERS(SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP) );  CREATE TABLE QRTZ_CRON_TRIGGERS   (     SCHED_NAME VARCHAR(120) NOT NULL,     TRIGGER_NAME VARCHAR(200) NOT NULL,     TRIGGER_GROUP VARCHAR(200) NOT NULL,     CRON_EXPRESSION VARCHAR(200) NOT NULL,     TIME_ZONE_ID VARCHAR(80),     PRIMARY KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP),     FOREIGN KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP)         REFERENCES QRTZ_TRIGGERS(SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP) );  CREATE TABLE QRTZ_SIMPROP_TRIGGERS   (               SCHED_NAME VARCHAR(120) NOT NULL,     TRIGGER_NAME VARCHAR(200) NOT NULL,     TRIGGER_GROUP VARCHAR(200) NOT NULL,     STR_PROP_1 VARCHAR(512) NULL,     STR_PROP_2 VARCHAR(512) NULL,     STR_PROP_3 VARCHAR(512) NULL,     INT_PROP_1 INT NULL,     INT_PROP_2 INT NULL,     LONG_PROP_1 BIGINT NULL,     LONG_PROP_2 BIGINT NULL,     DEC_PROP_1 NUMERIC(13,4) NULL,     DEC_PROP_2 NUMERIC(13,4) NULL,     BOOL_PROP_1 VARCHAR(1) NULL,     BOOL_PROP_2 VARCHAR(1) NULL,     PRIMARY KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP),     FOREIGN KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP)      REFERENCES QRTZ_TRIGGERS(SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP) );  CREATE TABLE QRTZ_BLOB_TRIGGERS   (     SCHED_NAME VARCHAR(120) NOT NULL,     TRIGGER_NAME VARCHAR(200) NOT NULL,     TRIGGER_GROUP VARCHAR(200) NOT NULL,     BLOB_DATA BLOB NULL,     PRIMARY KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP),     FOREIGN KEY (SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP)         REFERENCES QRTZ_TRIGGERS(SCHED_NAME,TRIGGER_NAME,TRIGGER_GROUP) );  CREATE TABLE QRTZ_CALENDARS   (     SCHED_NAME VARCHAR(120) NOT NULL,     CALENDAR_NAME  VARCHAR(200) NOT NULL,     CALENDAR BLOB NOT NULL,     PRIMARY KEY (SCHED_NAME,CALENDAR_NAME) );  CREATE TABLE QRTZ_PAUSED_TRIGGER_GRPS   (     SCHED_NAME VARCHAR(120) NOT NULL,     TRIGGER_GROUP  VARCHAR(200) NOT NULL,      PRIMARY KEY (SCHED_NAME,TRIGGER_GROUP) );  CREATE TABLE QRTZ_FIRED_TRIGGERS   (     SCHED_NAME VARCHAR(120) NOT NULL,     ENTRY_ID VARCHAR(95) NOT NULL,     TRIGGER_NAME VARCHAR(200) NOT NULL,     TRIGGER_GROUP VARCHAR(200) NOT NULL,     INSTANCE_NAME VARCHAR(200) NOT NULL,     FIRED_TIME BIGINT(13) NOT NULL,     PRIORITY INTEGER NOT NULL,     STATE VARCHAR(16) NOT NULL,     JOB_NAME VARCHAR(200) NULL,     JOB_GROUP VARCHAR(200) NULL,     IS_NONCONCURRENT VARCHAR(1) NULL,     REQUESTS_RECOVERY VARCHAR(1) NULL,     PRIMARY KEY (SCHED_NAME,ENTRY_ID) );  CREATE TABLE QRTZ_SCHEDULER_STATE   (     SCHED_NAME VARCHAR(120) NOT NULL,     INSTANCE_NAME VARCHAR(200) NOT NULL,     LAST_CHECKIN_TIME BIGINT(13) NOT NULL,     CHECKIN_INTERVAL BIGINT(13) NOT NULL,     PRIMARY KEY (SCHED_NAME,INSTANCE_NAME) );  CREATE TABLE QRTZ_LOCKS   (     SCHED_NAME VARCHAR(120) NOT NULL,     LOCK_NAME  VARCHAR(40) NOT NULL,      PRIMARY KEY (SCHED_NAME,LOCK_NAME) );   commit;   

编写相同的quartz.properties

#============================================================================ # Configure Main Scheduler Properties   #============================================================================  org.quartz.scheduler.instanceName = MyClusteredScheduler org.quartz.scheduler.instanceId = AUTO //每个集群自己的id  #============================================================================ # Configure ThreadPool   #============================================================================  org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 25  //这个大小可以不一致,其它都需要保持一致 org.quartz.threadPool.threadPriority = 5  #============================================================================ # Configure JobStore   #============================================================================  org.quartz.jobStore.misfireThreshold = 60000  org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate org.quartz.jobStore.useProperties = false org.quartz.jobStore.dataSource = myDS org.quartz.jobStore.tablePrefix = QRTZ_  org.quartz.jobStore.isClustered = true org.quartz.jobStore.clusterCheckinInterval = 20000  #============================================================================ # Configure Datasources   #============================================================================  org.quartz.dataSource.myDS.driver = oracle.jdbc.driver.OracleDriver org.quartz.dataSource.myDS.URL = jdbc:oracle:thin:@polarbear:1521:dev org.quartz.dataSource.myDS.user = quartz org.quartz.dataSource.myDS.password = quartz org.quartz.dataSource.myDS.maxConnections = 5 org.quartz.dataSource.myDS.validationQuery=select 0 from dual  

总结

quartz的分布式任务调度可以满足大多数的开发(虽然有一些限制),如果不满足可以了解一下当当网开源的任务调度框架,贴在参考部分。

参考

  1. quartz http://www.quartz-scheduler.org/documentation/quartz-2.1.x/configuration/ConfigJDBCJobStoreClustering

  2. 当当网开源 https://github.com/elasticjob/elastic-job

本文发表于2017年11月27日 00:37
(c)注:本文转载自https://my.oschina.net/tigerlene/blog/1580082,转载目的在于传递更多信息,并不代表本网赞同其观点和对其真实性负责。如有侵权行为,请联系我们,我们会及时删除.

阅读 2316 讨论 0 喜欢 1

抢先体验

扫码体验
趣味小程序
文字表情生成器

闪念胶囊

你要过得好哇,这样我才能恨你啊,你要是过得不好,我都不知道该恨你还是拥抱你啊。

直抵黄龙府,与诸君痛饮尔。

那时陪伴我的人啊,你们如今在何方。

不出意外的话,我们再也不会见了,祝你前程似锦。

这世界真好,吃野东西也要留出这条命来看看

快捷链接
网站地图
提交友链
Copyright © 2016 - 2021 Cion.
All Rights Reserved.
京ICP备2021004668号-1