Commit Graph

3188 Commits (7d7313fcc0ed6a7bb0a397c63b4bb7e03854f2df)

Author SHA1 Message Date
Yichao Yang 7d7313fcc0
[Fix][checkstyle] Fix checkstyle static type order checker (#3520) 2020-08-16 18:57:34 +08:00
Jave-Chen 42b0d67cef
add checkstyle job in github action. (#3518) 2020-08-16 12:37:26 +08:00
Yichao Yang fe9967b005
[Improvement-3369][api] Introduce ProjectService interface for more clear code (#3384)
* [Improvement][api] Introduce ProjectService interface for more clear code

* Fix ci e2e

* Update ProcessDefinitionServiceTest.java
2020-08-16 12:36:25 +08:00
rockxsj 2b46db0bf7
[Feature]add presto datasource support (#3308)
* * add presto datasource support
update .gitigonre to igonre some files

* * use another presto driver

* * add LICENSE files about presto-jdbc

* * just for test sonar

Co-authored-by: rockxsj <junjun@mgtv.com>
2020-08-14 08:52:31 +08:00
geosmart 6da7f1a2d1
resolve curator-client dependency confliction (#3488)
* dolphinscheduler-api.pom resolve curator-client conflict

the ds use curator-client:4.3.0 dependency by curator-frameworke,
but hadoop-common use  curator-client:2.7.1,
so make the ClassNotFoundException
```
Caused by: java.lang.ClassNotFoundException: org.apache.curator.connection.StandardConnectionHandlingPolicy
```
exclue curator-client in hadoop-common to resolve this

* resolve curator-client dependency confliction


## What is the purpose of the pull request

the ds use curator-client:4.3.0 dependency by curator-frameworke,
but hadoop-common use  curator-client:2.7.1,
so make the ClassNotFoundException
```
Caused by: java.lang.ClassNotFoundException: org.apache.curator.connection.StandardConnectionHandlingPolicy
```
exclue curator-client in hadoop-common to resolve this
2020-08-14 00:34:30 +08:00
zixi0825 46ab86d6df
batch copy or move process #2753 (#2884)
* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process sovle code smell

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* batch copy or move process

* Roll back the package.json file

* rerun test

* revert blank

* resolve conflicts

* resolve conflicts

* resolve ut error

* resolve sonar error

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-13 15:37:46 +08:00
zixi0825 a7a1156ff1
fix zk path error (#3470) 2020-08-12 17:16:43 +08:00
CalvinKirs d6a32ac652
[Feature#3234][cluster]enhanced load balancing (#3235)
* [Future#3234][cluster]enhanced load balancing
weight-based load balancing algorithm
this close # 3234

* remove useless parameter

* code smell

* load balancing according to work group

* add smooth weight round robin

* remove unused constants

* perfect unit test

* code smell

* code smell

* add work weight config

* fix config error

* add weight docs to readme.md
2020-08-10 19:05:21 +08:00
felix.wang 5584f0cb4d
[Fix-3262][common] When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled (#3264)
* fix bug #3165   get resource.storage.type value   toUpperCase

* fix bug #3176  optimize Gets the value of this property “resource.storage.type”

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix When you request the URL through applicationID to get the application status, you cannot get it if Kerberos authentication is enabled

* fix miss

* add KerberosHttpClient fix  kerberos bug

* fix map init

* add juint test

* Extraction of public methods

* Extraction of public methods

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* Fix code quality

* rebuild test

* rebuild test

* check style

* check style

* check style

* revert UT POM

* Kerberos judgment goes up to hadoopUtils

* fix merge

* Remove connection pool shutdown

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-10 15:18:01 +08:00
rockxsj 232a24441b
* add ZOOKEEPER_ROOT env in k8s deploy (#3260)
Co-authored-by: rockxsj <junjun@mgtv.com>
2020-08-10 15:16:19 +08:00
Yichao Yang db39f1f385
[Improvement][style] Improve checkstyle import order and code style importer (#3368)
* [Improvement][style] Add code import style automatic formatter

* Update checkstyle.xml

* Update checkstyle.xml

* [Improvement][api] Improve the code style and checkstyle of ds

* Update checkstyle.xml
2020-08-10 15:03:27 +08:00
Yichao Yang 133b4de129
[Improvement-3369][api] Introduce api service interface for ProcessDefinitionService (#3371)
* [Improvement-3369][api] Code clean and improvement

* Feature: import DATA_LIST static variable
2020-08-10 15:02:55 +08:00
Yichao Yang 379203b306
[Improvement-3369][api] Introduce session service interface for clear code (#3408)
* [Improvement][api] Introduce session service interface for clear code

* Update SessionService.java

* [Improvement][api] Add @Service

* Add licenses
2020-08-10 15:02:14 +08:00
Yichao Yang f419fe6239
[Test][server] Add LoggerServerTest (#3405) 2020-08-10 15:01:20 +08:00
BoYiZhang 8bec5eb639
[Feature][Queue]`Queue manage ' is renamed' yarn queue manage ' (#3430)
* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* dag  add close button

* reset last version

* reset last version

* dag add close buttion

dag add close buttion

* update  CLICK_SAVE_WORKFLOW_BUTTON  xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* Update CreateWorkflowLocator.java

modify submit workflow button

* Update CreateWorkflowLocator.java

* Update CreateWorkflowLocator.java

modify CLICK_ADD_BUTTON

* Update CreateWorkflowLocator.java

delete print

* Update CreateWorkflowLocator.java

1

* Update CreateWorkflowLocator.java

1

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dag.vue

* Update dag.vue

* Update dag.vue

* Update CreateWorkflowLocator.java

* Revert "Setting '-XX:+DisableExplicitGC ' causes netty memory leaks"

This reverts commit 3a2cba7a

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dolphinscheduler-daemon.sh

* `Queue manage` is renamed ` yarn queue manage `

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-08-10 14:50:08 +08:00
Yichao Yang 3ed0afa6ac
[Improvement-3369][api] Introduce data analysis service interface for clear code (#3439)
* [Improvement][api] Introduce data analysis service interface for clear code

* Remove the bad smell

* Remove the bad smell
2020-08-10 14:47:45 +08:00
Yichao Yang dce403d062
[Improvement-3369][api] Introduce logger service interface for clear code (#3437)
* [Improvement][api] Introduce logger service interface for clear code

* Remove the code smell
2020-08-10 14:47:16 +08:00
Yichao Yang f7f07608ed
[Improvement][api] Introduce access token service interface for clear code (#3438) 2020-08-10 14:46:59 +08:00
Yichao Yang 4fb06a736d
[Test][api] Introduce api controller unit test example (#3447)
* [Test][api] Introduce api controller test example

* Add controller test
2020-08-10 14:36:31 +08:00
zixi0825 40a21816ab
[Fix] [Server] Fix the spell error and add log in SqlTask (#3243)
* fix the spell and modify logger print format
2020-08-07 13:58:14 +08:00
wuchunfu f784ce504f
[Fix-3256][ui] Fix admin user info update error (#3425)
* [PROPOSAL-3139] Datasource selection changes from radio to select

* [PROPOSAL-3139] Datasource selection changes from radio to select

* [BUG FIX] issues #3256

* [BUG FIX] issues #3256

* [BUG FIX] issues #3256

* [Fix-3256][ui] Fix admin user info update error

* [Fix-3256][ui] Fix admin user info update error

* [Fix-3256][ui] Fix admin user info update error

* [Fix-3256][ui] Fix admin user info update error

* reset createUser.vue

* [Fix-3256][ui] Fix admin user info update error

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-07 10:20:56 +08:00
JinyLeeChina 2b990e1f00
Update actions.js (#3401) 2020-08-05 18:59:49 +08:00
sky 6dc500915c
[Feature-3392][api-server] (#3403)
* feature user register

fix bug

fix security problem

fix security problem

* activate user

* fix confilct

* fix confilct and fix some bug

* fix cr problem

Co-authored-by: dev_sky <dev_sky@740051880@qq.com>
2020-08-05 17:27:55 +08:00
CalvinKirs 6ecc95a338
[FIX_BUG][*] fix some bugs from sonar analysis (#3322)
* [FIX_BUG][*] fix some bugs from sonar analysis

* remove invalid single test

* code smell

* remove exception catches that never happen

* delete invalid exception catch

* remove excess locks

* fix e2e test error

* recovery related exception removal
2020-08-04 02:23:50 +00:00
wuchunfu 27b296c45b
[Fix-3256][ui] Fix admin user info update error (#3306)
* [PROPOSAL-3139] Datasource selection changes from radio to select

* [PROPOSAL-3139] Datasource selection changes from radio to select

* [BUG FIX] issues #3256

* [BUG FIX] issues #3256

* [BUG FIX] issues #3256

* [Fix-3256][ui] Fix admin user info update error

* [Fix-3256][ui] Fix admin user info update error

* [Fix-3256][ui] Fix admin user info update error

* [Fix-3256][ui] Fix admin user info update error

* reset createUser.vue

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-08-03 13:50:30 +08:00
BoYiZhang d70d8a6054
[Feature-3222][datasource] Store password in ciphertext instead (#3330)
* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* fix bug

Delete invalid field: executorcores

Modify verification prompt

* dag  add close button

* reset last version

* reset last version

* dag add close buttion

dag add close buttion

* update  CLICK_SAVE_WORKFLOW_BUTTON  xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* updae CLICK_SAVE_WORKFLOW_BUTTON xpath

* Update CreateWorkflowLocator.java

modify submit workflow button

* Update CreateWorkflowLocator.java

* Update CreateWorkflowLocator.java

modify CLICK_ADD_BUTTON

* Update CreateWorkflowLocator.java

delete print

* Update CreateWorkflowLocator.java

1

* Update CreateWorkflowLocator.java

1

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Update dag.vue

* Update dag.vue

* Update dag.vue

* Update CreateWorkflowLocator.java

* Revert "Setting '-XX:+DisableExplicitGC ' causes netty memory leaks"

This reverts commit 3a2cba7a

* Setting '-XX:+DisableExplicitGC ' causes netty memory leaks

in addition

update '- XX: largepagesizeinbytes = 128M' to '- XX: largepagesizeinbytes = 10M'

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* Store password in ciphertext instead

* update test unit

* add test unit

* add test unit

* update e2e test

* update unit test

* update unit test

Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
2020-08-01 14:28:12 +08:00
dailidong 6d903e266f
[Feature][server] add embeded zookeeper server in dolphinscheduler-daemon.sh (#3293)
* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

* simplify common config

* add worker group config

* add worker group config

* add worker group config

* this command just for getting a quick experience,not recommended for production. this operation will start a standalone zookeeper server

* Merge remote-tracking branch 'upstream/dev' into dev

* add logback for zookeeper

* Update logback-zookeeper.xml

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
2020-07-31 17:37:02 +08:00
xsbai 87eb7c8211
adds waterdrop plugin (#3073)
* add plugin waterdrop

* add plugin waterdrop

* add plugin waterdrop

* fix deploy mode error

* Update en_US.js

fix conflict

* Update en_US.js

fix conflict

* Update en_US.js

Add missing commas

* Update zh_CN.js

Add missing commas

* Update waterdrop.vue

Fix the no resource hint

* Update zh_CN.js

Add 'Please select the waterdrop resources'

* Update en_US.js

Add  'Please select the waterdrop resources'

* Update waterdrop.vue

fix bug when click the waterdrop node again, it still show the `client` mode in deploy mode

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 23:20:59 +08:00
RedemptionC 0c5754c3f2
add feature : query top n success process instance order by running duration (#3315)
* add feature : query top n success process instance order by running duration

* add feature : add params check

* add feature : remove code smell

* add feature : remove code smell

* add feature : add param check

* add feature : add param check
improve handleEscape method in ParameterUtils.java

* top n : add  java doc and format code
2020-07-30 18:38:59 +08:00
zixi0825 5fe31546ba
[Fix][all] fix spell error (#3289)
* fix spell error

* test unit test

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-30 18:33:16 +08:00
zhuangchong 583a615519
[Improvement][install] The new shell script is used to view the server processes of nodes #3347 (#3348)
* dev 开发环境变量

* add status-all.sh ,used to view the server processes of nodes.

* update.

* update.

* update.

* add status-all.sh ,used to view the server processes of nodes.

Co-authored-by: unknown <zhuangchong-phq@sinosig.com>
2020-07-30 18:30:57 +08:00
chendapao afd9139c3f
[Feature]use zk own del func (#3310)
* use zk own del func

* Update remove-zk-node.sh

Co-authored-by: dailidong <dailidong66@gmail.com>
2020-07-30 18:26:27 +08:00
Yichao Yang d006bd3ba5
[Improvement][api] Refactor the transaction rollbackfor runtimeexception (#3313)
* [Improvement][api] Refactor the transaction rollbackfor runtimeexception

* Remove the unused import
2020-07-30 15:12:14 +08:00
Yichao Yang d79af50809
[Fix-3077][ui] Fix the edit name duplicated verify (#3346) 2020-07-30 15:04:57 +08:00
LiemLin f27c62a5db
workflow lineage (#2421)
* workflow lineage

* workflow lineage unit test

* stash

* add apache license

* Update .env

* code optimize

* add unit test

* add apache license

* add apache license

* optimized code

* add postgresql support and optimized code

* Update pom.xml

* Update en_US.js

* Update zh_CN.js

* remove mock code

* inport i18n

* Update pom.xml

* Delete .env

* add env file

Co-authored-by: linhaiqiang <haiqiang.lin@five-star.cn>
Co-authored-by: dailidong <dailidong66@gmail.com>
Co-authored-by: becarchal <yangbin_57@sina.com>
2020-07-30 13:32:29 +08:00
CalvinKirs 3a0d4da27e
[Test][server] Fix Test ContextConfiguration error. (#3304)
* [FIX_BUG]fix test error

* small change

* add test

* reformat

* fix test no qualifying bean
2020-07-30 13:28:27 +08:00
felix.wang b555d0a086
[Improvement][common] HttpClient singleton mode #3332 (#3350)
* Optimize the httpclient

* Optimize the httpclient singleton

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java

* Merge branch 'github-local/dev' into feature/httpClientsingleton

# Conflicts:
#	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HttpUtils.java
#	dolphinscheduler-common/src/test/java/org/apache/dolphinscheduler/common/utils/HttpUtilsTest.java
2020-07-30 13:24:54 +08:00
Yichao Yang a532d8ed69
[Fix-3338][ui] Fix the warning when user do not fill the node blank (#3349)
* [Fix-3338][ui] Fix the warning when user do not fill the node blank

* Update formModel.vue
2020-07-30 13:14:05 +08:00
zixi0825 78856fb08e
remove the duplicate function in MasterExecThread (#3329)
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2020-07-29 19:15:03 +08:00
Yichao Yang afeeb34d2e
[Improvement][style] Add code import style automatic formatter (#3333) 2020-07-29 19:13:17 +08:00
felix.wang e026d303ea
Optimize the httpclient (#3337) 2020-07-29 19:03:44 +08:00
谌启成 e2a117032d
[Fix-3295][api] Add default value for request RequestParam (#3341) 2020-07-29 19:01:53 +08:00
xingchun-chen 5f725aa28b
Merge pull request #3331 from lenboo/dev-complement
[FIX-1336] [master] serial complement data can have multiple process instances
2020-07-29 15:37:49 +08:00
lenboo 6c2e7d74d1 merge from dev 2020-07-28 19:02:39 +08:00
lenboo 82295eea52 fix bug#1336 set start/end time before complement the next process 2020-07-28 17:57:49 +08:00
bao liang 624f0aeab9
fix bug#1336 serial complement data can have multiple process instances (#3317)
Co-authored-by: lenboo <baoliang@analysys.com.cn>
2020-07-28 11:12:51 +08:00
lenboo 39182568b2 fix bug#1336 serial complement data can have multiple process instances 2020-07-27 15:43:08 +08:00
Yichao Yang 0577dc9780
[Test-3288][server] Fix github ci unit test oom (#3305)
* [Improvement] Test github action oom

* Optimize the test case performance

* Remove the unused import
2020-07-25 19:24:16 +08:00
shi-hailong 8d70c3b7c1
Update dolphinscheduler.conf (#3296)
/ui下的资源访问不应该发送到api-server
2020-07-24 17:09:58 +08:00
tswstarplanet 5b7efd2d33
[Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client (#3244)
* [Feature-3134]Refactor to use a shared singleton Curator Zookeeper Client

* autowire CuratorZookeeperClient to ZookeeperOperator

* [Improvement] Add the issue specifications reference (#3221)

* log zookeeper address when the connect state change

* resume the operation of add connect state listener in MasterRegistry

Co-authored-by: Yichao Yang <1048262223@qq.com>
2020-07-23 11:02:30 +08:00