disjob

A distributed task scheduling and distributed computing framework

336

Java

A distributed task scheduling and distributed computing framework

English | 简体中文

A distributed task scheduling and distributed computing framework, in addition to having regular task scheduling functions, also provides: pausing/cancelling running tasks, resuming paused tasks, task splitting, failure retries, broadcast tasks, task dependencies, workflow tasks (DAG), separate deployment of managers and executors, Web management backend, and other capabilities.

Lightweight and easy to use, it is particularly suitable for the execution of long tasks. Powerful and reliable, it has been tested in production environments.

Architecture

Overall Process

Overall Process

Code Structure

disjob                                        # Main project①
├── disjob-admin                              # Management backend project② (Based on RuoYi framework secondary development)
├── disjob-bom                                # Maven project bom module
├── disjob-common                             # Common utility class module
├── disjob-core                               # Task scheduling related core classes (such as data models, enumeration classes, abstract layer interfaces, etc.)
├── disjob-dispatch                           # Task dispatch module
│   ├── disjob-dispatch-api                   # Abstract interface layer of task dispatch
│   ├── disjob-dispatch-http                  # Http implementation of task dispatch
│   └── disjob-dispatch-redis                 # Redis implementation of task dispatch
├── disjob-id                                 # Distributed ID generation module
├── disjob-registry                           # Server (Supervisor & Worker) registration module
│   ├── disjob-registry-api                   # Abstract interface layer of Server registration center
│   ├── disjob-registry-consul                # Server registry center: Consul implementation
│   ├── disjob-registry-database              # Server registry center: Database implementation
│   ├── disjob-registry-etcd                  # Server registry center: Etcd implementation
│   ├── disjob-registry-nacos                 # Server registry center: Nacos implementation
│   ├── disjob-registry-redis                 # Server registry center: Redis implementation
│   └── disjob-registry-zookeeper             # Server registry center: Zookeeper implementation
├── disjob-reports                            # Aggregate test coverage reports of various modules
├── disjob-samples                            # Samples project③
│   ├── disjob-samples-conf-common            # Samples common config (log4j2.xml)
│   ├── disjob-samples-conf-supervisor        # Samples Supervisor config
│   ├── disjob-samples-conf-worker            # Samples Worker config
│   ├── disjob-samples-frameless-worker       # Example of Worker deployed separately (ordinary Java-main application)
│   ├── disjob-samples-springboot-common      # Samples Spring-boot common module
│   ├── disjob-samples-springboot-merged      # Example of Supervisor and Worker deployed together (Spring-boot application)
│   ├── disjob-samples-springboot-supervisor  # Example of Supervisor deployed separately (Spring-boot application)
│   └── disjob-samples-springboot-worker      # Example of Worker deployed separately (Spring-boot application)
├── disjob-supervisor                         # Supervisor code
├── disjob-test                               # Used for auxiliary testing
└── disjob-worker                             # Worker code

Features

Divided into two roles: Manager (Supervisor) and Executor (Worker), Supervisor and Worker can be deployed separately
Supervisor and Worker discover each other through the registry center, supported: Database, Redis, Consul, Nacos, Zookeeper, Etcd
Supervisor is responsible for generating tasks and dispatching them to Worker for execution, supported: Redis, Http
Need to specify the group, Job tasks will only be dispatched to the specified group of Workers for execution
Provides the ability to split tasks, override the method JobSplitter#split to split many tasks, then distributed and parallel execution
Supports pausing and cancelling running tasks, paused tasks can be resumed for execution, failed tasks support retry
Supports savepoint task execution snapshot, so that manually or abnormally paused tasks can be resumed from the savepoint
If a task throw PauseTaskException at executing, then will pause all instance tasks (even if dispatched other worker machine tasks)
Supports broadcast tasks, broadcast tasks will be dispatched to all workers under the group for execution
Supports dependencies jobs, multiple Jobs configured with dependencies will be executed in the established dependency order
Supports DAG workflows, can configure jobExecutor as a complex DAG expression, such as: A->B,C,(D->E)->D,F->G
Provides a Web management backend, job configuration, task monitoring, etc.

Download From Maven Central

<dependency>
  <groupId>cn.ponfee</groupId>
  <artifactId>disjob-{xxx}</artifactId>
  <version>2.1.5</version>
</dependency>

Build From Source

./mvnw clean install -DskipTests -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -U

Quick Start

Import the projects to IDE (three independent projects all in one git repository)

Primary project: disjob
Admin project: disjob-admin
Sample project: disjob-samples

Start the following applications to form a distributed scheduling cluster

Configured different ports, run the java main class in IDE, or run the built jar package with the java -jar command.

Start the embedded local Mysql & Redis
- Refer to this document if reports MariaDB “Library not loaded” error on MacOSX
- When connecting to mysql with a client tool, the username is root, no password is required
- The initialization SQL script has been automatically executed at startup
Start the Worker Java-main application
Start the Worker Spring-boot application
Start the Supervisor+Worker Spring-boot application
Start the Supervisor Spring-boot application
Start the Admin Spring-boot application

Open【 http://127.0.0.1:80/ 】in your browser and login with username/password: admin/123456
Find the Scheduling Management menu in the left sidebar to use the scheduling management function
Scheduling Configuration: Job configuration, including viewing, adding, modifying, deleting, triggering, disabling, etc.
Scheduling Instances: job trigger time generate an instance, the instance split to many tasks.

Link Address

Admin demo:【 http://ponfee.cn:8000/ 】, username/password: disjob/123456
Online documentation: Under construction, please stay tuned!

User Guide

The project includes two SQL scripts

Change the specific implementation of disjob-registry-{xxx} and disjob-dispatch-{xxx} in the maven pom file

Samples project
Admin project
By default, disjob-registry-redis is used as the server registry and disjob-dispatch-http is used for task dispatch

Samples project configuration files

Admin project configuration files

Reference for various registry configuration classes (Redis uses the configuration method provided by Spring-boot)

For example, if using Consul as the registry center, you can add the following configuration

disjob.registry.consul:
  namespace: disjob_registry
  host: localhost
  port: 8500
  token:

Some embedded local servers

Contributing

If you find bugs, have better implementation solutions, new features and so on, you can submit a PR or create newIssues

Communication

Scan the QR code to add WeChat friends, note disjob, and invite you to join the group WeChat.

TODO List

[x] Extend registry: Zookeeper, Etcd, Nacos
[x] Workflow tasks (DAG)
[x] Admin console: accounting, authority, permission and task manage
[ ] Build a project document web site
[ ] Monitor real-time executing logs of tasks online
[ ] Alarm subscribe: Email, SMS, Voice, Lark, Ding Talk, WeChat
[ ] Visual monitoring BI dashboard