7.0 KiB

Raw Blame History

DolphinScheduler development

Software Requests

Before setting up the DolphinScheduler development environment, please make sure you have installed the software as below:

Git: DolphinScheduler version control system
JDK: DolphinScheduler backend language
Maven: Java Package Management System
Node: DolphinScheduler frontend language

Clone Git Repository

Download the git repository through your git management tool, here we use git-core as an example

mkdir dolphinscheduler
cd dolphinscheduler
git clone git@github.com:apache/dolphinscheduler.git

compile source code

i. If you use MySQL database, pay attention to modify pom.xml in the root project, and change the scope of the mysql-connector-java dependency to compile.

ii. Run mvn clean install -Prelease -Dmaven.test.skip=true

Notice

There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode

Standalone mode: Recommended，more convenient to build development environment, it can cover most scenes.
Normal mode: Separate server master, worker, api, which can cover more test environments than standalone, and it is more like production environment in real life.

DolphinScheduler Standalone Quick Start

Note: Standalone server only for development and debugging, cause it use H2 Database, Zookeeper Testing Server which may not stable in production Standalone is only supported in DolphinScheduler 1.3.9 and later versions

Git Branch Choose

Use different Git branch to develop different codes

If you want to develop based on a binary package, switch git branch to specific release branch, for example, if you want to develop base on 1.3.9, you should choose branch 1.3.9-release.
If you want to develop the latest code, choose branch branch dev.

Start backend server

Find the class org.apache.dolphinscheduler.server.StandaloneServer in Intellij IDEA and clikc run main function to startup.

Start frontend server

Install frontend dependencies and run it

cd dolphinscheduler-ui
npm install
npm run start

The browser access address http://localhost:12345/dolphinscheduler can login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123

DolphinScheduler Normal Mode

Prepare

zookeeper

Download ZooKeeper, and extract it.

Create directory zkData and zkLog
Go to the zookeeper installation directory, copy configure file zoo_sample.cfg to conf/zoo.cfg, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper
```
# We use path /data/zookeeper/data and /data/zookeeper/datalog here as example
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/datalog
```
Run ./bin/zkServer.sh in terminal by command ./bin/zkServer.sh start.

Database

The DolphinScheduler's metadata is stored in relational database. Currently supported MySQL and Postgresql. We use MySQL as an example. Start the database and create a new database named dolphinscheduler as DolphinScheduler metabase

After creating the new database, run the sql file under dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql directly in MySQL to complete the database initialization

Start Backend Server

Following steps will guide how to start the DolphinScheduler backend service

Backend Start Prepare

Open project: Use IDE open the project, here we use Intellij IDEA as an example, after opening it will take a while for Intellij IDEA to complete the dependent download
Plugin installation(Only required for 2.0 or later)
Registry plug-in configuration, take Zookeeper as an example (registry.properties) dolphinscheduler-service/src/main/resources/registry.properties

 registry.plugin.name=zookeeper
 registry.servers=127.0.0.1:2181

File change
- If you use MySQL as your metadata database, you need to modify dolphinscheduler/pom.xml and change the scope of the mysql-connector-java dependency to compile. This step is not necessary to use PostgreSQL
- Modify database configuration, modify the database configuration in the dolphinscheduler-dao/src/main/resources/application-mysql.yaml
We here use MySQL with database, username, password named dolphinscheduler as an example
```
 spring:
   datasource:
     driver-class-name: com.mysql.jdbc.Driver
     url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
     username: ds_user
     password: dolphinscheduler
```
Log level: add a line <appender-ref ref="STDOUT"/> to the following configuration to enable the log to be displayed on the command line

dolphinscheduler-server/src/main/resources/logback-worker.xml

dolphinscheduler-server/src/main/resources/logback-master.xml

dolphinscheduler-api/src/main/resources/logback-api.xml

here we add the result after modify as below:
```
<root level="INFO">
+  <appender-ref ref="STDOUT"/>
  <appender-ref ref="APILOGFILE"/>
  <appender-ref ref="SKYWALKING-LOG"/>
</root>
```

Note: Only DolphinScheduler 2.0 and later versions need to inatall plugin before start server. It not need before version 2.0.

Server start

There are three services that need to be started, including MasterServer, WorkerServer, ApiApplicationServer.

MasterServer：Execute function main in the class org.apache.dolphinscheduler.server.master.MasterServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false -Dspring.profiles.active=mysql
WorkerServer：Execute function main in the class org.apache.dolphinscheduler.server.worker.WorkerServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false -Dspring.profiles.active=mysql
ApiApplicationServer：Execute function main in the class org.apache.dolphinscheduler.api.ApiApplicationServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-api.xml -Dspring.profiles.active=api,mysql. After it started, you could find Open API documentation in http://localhost:12345/dolphinscheduler/doc.html

The mysql in the VM Options -Dspring.profiles.active=mysql means specified configuration file

Start Frontend Server