Commit Graph

942 Commits (0d0674df1f78d666fc7c2155629627380aa0a9b1)

Author SHA1 Message Date
caishunfeng 994de4c902
[Bug-9045][Server] improve log detail error if task instance host is null (#9205)
* improve log detail error if task instance host is null

* test

Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 11:36:56 +08:00
caishunfeng 263791a63e
[Bug-9137][Server] fix timezone init (#9219)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-28 09:38:51 +08:00
nobolity 0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
calvin 7c5bebea98
[Fix-8544][API] The folder's size can't be updated when creating, updating or deleting a resource file int the folder. (#9107) 2022-03-23 18:58:41 +08:00
Tq 3c377ff54a
[Bug-9094][API] fix default time-zone conf (#9095)
* fix the API server default time-zone conf

* fix the Python server default time-zone conf
2022-03-22 19:38:46 +08:00
calvin 9b163249d6
[Fix-9025] [UI Next] Rectify this issue about failing to save a process when the user is ordinary user. (#9063)
* modify the back-edn api

* fix this issue
2022-03-21 21:52:07 +08:00
yangyunxi b7cf8b49c5
AlertPluginInstanceController verifyGroupName method code optimizatio… (#8920)
* AlertPluginInstanceController verifyGroupName method code optimization #8863
Co-authored-by: Kerwin <37063904+zhuangchong@users.noreply.github.com>
2022-03-16 10:42:03 +08:00
caishunfeng 60fbbcb790
[Feature-8530][API] add system default timezone if no user timezone when get userInfo (#8899)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-15 17:46:41 +08:00
Kerwin a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
calvin c7e80e42d7
[Fix-8836][UI Next][V1.0.0-Alpha] Rectify this issue about creating a file or directory with a wrong path. (#8892)
* fix this issue

* fix this issue

* fix code style

* fix code style
2022-03-15 14:19:19 +08:00
songjianet db64dcade8
Revert "[Fix-8836][UI Next][V1.0.0-Alpha] Rectify this issue about creating a file or directory with a wrong path. (#8880)" (#8888)
This reverts commit e4dcf3d2e0.
2022-03-15 10:13:37 +08:00
calvin e4dcf3d2e0
[Fix-8836][UI Next][V1.0.0-Alpha] Rectify this issue about creating a file or directory with a wrong path. (#8880)
* fix this issue

* fix this issue
2022-03-15 09:35:14 +08:00
Tq 1f83140558
fix create&update wrokflow cron APIs, verify end time must bigger than start time (#8871) 2022-03-14 16:40:07 +08:00
kezhenxu94 d3cbf8eeaf
[UI-NEXT] Fix wrong packaging locations (#8851) 2022-03-14 15:26:15 +08:00
ouyangyewei 5fa1c4e7e3
[Feature][dolphinscheduler-api] Return domain object after creating success (#8825)
Co-authored-by: ouyangyewei <yewei.oyyw@alibaba-inc.com>
2022-03-13 21:20:36 +08:00
zhanqian e466f54c5d
issue 8645 (#8811)
Co-authored-by: 北笙 <“zhanqian@cai-inc.com”>
Co-authored-by: zhanqian <zhanqian@cai-inc.com>
2022-03-10 20:01:44 +08:00
Sunny Lei 2aed250ed4
[DS-7896][fix] Fixed the problem of duplicating project names when authorizing projects (#8453)
This closes #7896
2022-03-10 16:58:02 +08:00
caishunfeng ddd4474189
change quartz executor to be managed by spring (#8774)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-03-09 21:36:38 +08:00
xiangzihao fc334b7b13
[DS-8580] Fix ui-next serve router for `/` path (#8748) 2022-03-08 11:09:08 +08:00
kezhenxu94 d2a360480a
[CI][UI NEXT] Package new UI into release tar and remove old UI from it (#8670)
* [CI][UI NEXT] Package new UI into release tar and remove old UI from it

* Update e2e.yml
2022-03-07 11:20:24 +08:00
xiangzihao 2ab8c1ca7d
[Fix-8616][WorkFlowLineage] work flow lineage search result missing data (#8684)
* fix bug_8616

* remove meaningless query column

* fix code smell

* fix remaining problems
2022-03-04 14:05:59 +08:00
kezhenxu94 e04bdba22f
Adapt web configurations for UI next (#8639) 2022-03-02 12:10:38 +08:00
xiangzihao 3cfb3270fe
[Fix-8581][API-Doc] Add some missing chinese translation (#8588) 2022-03-01 15:14:52 +08:00
yimaixinchen a72742b9de
remove unuse import jsonFormat (#8548) 2022-02-25 22:05:05 +08:00
xiangzihao 891a002bea
[Feature-#8373][MasterServer] Dependent tasks can re-run automatically in the case of complement (#8496)
* first add feature_8373

* fix code smell

* add blank line

* fix some problems

* fix unit test error
2022-02-25 10:34:02 +08:00
caishunfeng 84f2cfec39
[Feature][Server] Custom timezone - Add timezone handler unified (#8477)
* date convert of timezone

* remove @JsonFormat

* add unit test

* fix time preview in scheduler

* optimization & add env config

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-24 18:20:34 +08:00
Sunny Lei 1ffb5d6e8d
[DS-7896][fix] Fixed when a non-admin user creates a udf function, resources other than .jar will be displayed in the UDF resource (#8458)
This closes #7892
2022-02-24 18:17:25 +08:00
kezhenxu94 fa19006a18
Add a property to enable/disable metrics module (#8484) 2022-02-24 10:21:47 +08:00
calvin c590ad43d7
fix this issue (#8509) 2022-02-24 08:50:03 +08:00
KingsleyY 2a844dcc67
[Improvement][Api] add @Override on the Override method (#8452) 2022-02-22 13:10:28 +08:00
天仇 95c119226b
[Feature-8386][Server] Custom timezone - Add timezone into user info (#8457)
* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Add system default timezone like `GMT+08:00` when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Add system default timezone like `GMT+08:00` when create user
  3. Add timeZone property when update user info
  4. add ddl sql in update sql script

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

 1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

  1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386

* [Feature-8386][Server] Custom timezone - Add timezone into user info

1. Add column `time_zone varchar(32) default null` in table `t_ds_user`
  2. Set timeZone null when create user
  3. Add timeZone property when update user info

This completes #8386
2022-02-21 23:28:31 +08:00
Kirs 0ef49b4a98
[Improve]Use hibernate validation framework (#4) (#8467)
* [Improve]Use hibernate validation framework
Improve email verification method
Support domain name verification
* Email is at least a second-level domain name
2022-02-21 16:54:46 +08:00
JinYong Li 8d0a0f9c32
[Bug] [API] fix releaseWorkflowAndSchedule api and relation binding api (#8464)
* fix relation unbinding bug

* fix 8463
2022-02-21 15:58:59 +08:00
JinYong Li f9c090e048
fix relation unbinding bug (#8379) 2022-02-17 15:38:13 +08:00
caishunfeng 71fcea5937
fix empty upstreamTaskCodes (#8405)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-16 23:06:32 +08:00
qianli2022 151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 (#8252)
* add db operator for namespace

* add k8s client

* add k8s controller

* add namespace ui

* add licenses

* add kubernetes model known-dependencies

* add new licenses

* fix SonarCloud Code Analysis bug

* remove license for standard protocol

* fix variable MemoryStr to memoryStr

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-02-15 10:21:15 +08:00
JinYong Li 4a4e72b0d2
pick 8270/8308 (#8355) 2022-02-14 12:03:12 +08:00
EdwardYang 13d9031b68
[Bug][API] fix ProcessDefinition update error #8334 (#8338)
* [bug] fix task definition delete

* [bug] fix ProcessDefinition update error #8334

* [bug] fix ProcessDefinition update error #8334

Co-authored-by: edward-yang <yangjianh210@gmail.com>
2022-02-11 14:59:06 +08:00
ouyangyewei 294882f5d7
[Improvement-8218][dolphinscheduler-api] QueryTaskInstanceListPaging API performance improvement (#8223) 2022-02-09 12:58:15 +08:00
JinYong Li 65744d4624
[Cherry-pick-dev][API] fix relation api bug (#8207)
* pick 8111

* fix relation delete (#8190)

* pick 8190
2022-01-28 13:32:22 +08:00
zixi0825 4ba6d667fc
[Feature][DataQuality] Add Data quality Module #4283 (#6718)
* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA

* fix Constants bug

* remove the unused class

* add unit test

* fix code style error

* add unit test

* refactor data quality common entity

* fix code style error

* add unit test

* close e2e test

* fix code smell bug

* modify dataquality enum value to 14 in TaskType

* add data qualtiy task

* update

* add getDatasourceOptions interface

* fix checkstyle

* close e2e test

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* add unit test and alter dataquality task result

* fix checkstyle

* fix process service test error

* add unit test and fix code smells

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* change execute sql type name

* revert ui pom.xml

* fix data quality task error

* fix checkstyle error

* fix dq task src_connector_type ui select bug

* fix spark rw postgresql bug

* change mysql driver scope

* fix form-create json bug

* fix code smell

* fix DolphinException Bug

* fix ui validate rule and Alert title

* fix target connection param bug

* fix threshold validate change

* add rule input entry index

* change statistic_comparison_check logic

* remove check type change

* add DateExpressionReplaceUtil

* fix null point expetion

* fix null point expetion

* fix test error

* add more sql driver

* fix test error and remove DateExprReplaceUtil

* add get datasource tables and columns

* add get datasource tables and columns

* remove hive-jdbc in pom.xml

* fix code smells

* update sql

* change the pom.xml

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

* update sql

* [Feature][DataQuality] Add data quality task ui (#5054)

* add data quality task ui

* update skywalking-eyes SHA

* fix style

* fix eslint error

* fix eslint error

* test e2e

* fix dq task src_connector_type ui select bug

* fix threshold validate change

* remove check type change

* add get datasource tables and columns

* optimize multi_table_accuracy ui

* fix v-show error

* fix code smells

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality module  (#4830)

* add data quality module

* add license

* add package configuration in dist pom

* fix license and jar import bug

* replace apache/skywalking-eyes@9bd5feb SHA

* refacotr jbdc-connector and writer

* modify parameter name in HiveConnector

* fix checkstyle error

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix checkstyle error in dolphinschesuler-dist

* fix duplicate code bug

* fix code style bug

* fix code smells

* update

* close e2e test

* fix spark rw postgresql bug

* change mysql driver scope

* add more sql driver

* remove hive-jdbc in pom.xml

* change the pom.xml

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* [Feature][DataQuality] Add data quality task backend (#4883)

* add dq relevant enums and parameter

* replace apache/skywalking-eyes@9bd5feb SHA


Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

* refactor data_quality_module

* add header license

* data quality module refactor

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix checkstyle error

* fix unit test error

* fix code smell

* fix check style

* fix unit test error

* task statistics value add unique code

* fix unit test error

* fix checkstyle error

* fix checkstyle

* fix security hotspot

* fix unit test error

* fix security hotspot

* fix check

* add data quality task error handling

* fix unit test error

* add unit test

* add unit test

* optimize data quality result alert

* fix unit test

* fix sql script error

* fix bug

* update sql script

* fix checkstyle

* add license

* fix checkstyle

* fix checkstyle

* fix unit test

* add jacoco dependencies

* fix unit test

* fix unit test

* add jacoco dependencies

* add unit test

* add unit test

* add license

* fix checkstyle

* fix pom

* fix checkstyle

* fix checkstyle

* merge dev

* fix ui error

* fix pom error

* fix pom error

* fix test error

* fix test error

* mssql-jdbc exclude azure-keyvault

* fix test error

* merge dev and add unit test

* add notes

* rollback the CollectionUtils

* fix

* update sql

* fix

* fix

* fix query rule page error

* change dq.jar path

* fix sql error

* fix ui error

* fix(dq): jar path&task enum description

* add notes on DataQualityApplication

* fix dq result jump error

* fix(ui): page condition

* feat(ui): add show error output path

* change version

* remove all chinese word in sql

* merge

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
2022-01-27 15:55:45 +08:00
yc322 fb9f4a1339
[Feature-5822][api] Add audit log (#6322)
* rebase

* update audit log type name

* fix the quoted property error

* fix bug

* update audit global switch default value

* fix wrong ddl comment

* fix wrong code style

* update audit log search sql

* add licenses

* add unit test

* fix wrong code style

* fix bugs

* fix wrong code style

* add test case

* add test case

* add license

* Add unit test in pom.xml

* resolve code smell problem

* mysql and postgresql ddl update

* update audit log path and audit page query

* update audit configuration

* update audit log schema, add resource type and resource id

* update audit query and list page

* resolve audit log front page bugs and log query error

* update audit configuration

* resolve wrong configuration

* update AuditLogControllerTest

* update application.yaml

* resolve code smell

* update standalone configuration

* resolve bugs

Co-authored-by: yc322 <1393242780@qq.com>
2022-01-26 09:53:18 +08:00
Kerwin 103ccb01da
change version to 2.0.4-SNAPSHOT (#8188) 2022-01-25 15:17:17 +08:00
xiangzihao b5631d5281
[Fix-8119][API] Update ProcessInstance error and without reason output (#8122)
* fix bug_8119

* fix bug_8119

* fix bug_8119

* fix bug_8119

* fix bug_8119
2022-01-21 09:32:38 +08:00
XiaochenNan 3456904af8
add unit test of queryWorkAddressLest (#8054)
Co-authored-by: brian <flyunrain@163.com>
2022-01-17 13:36:46 +08:00
calvin 7391cc20f0
[Fix-8056] Rectify this issue about failing to edit worker groups for an environment. (#8072) 2022-01-17 10:30:40 +08:00
wind c0d69125eb
[2.0.3][Bug-7986] Disable deletion of running workflow (#8065)
Co-authored-by: caishunfeng <534328519@qq.com>
2022-01-17 10:29:59 +08:00
calvin 5bcf7bcce6
fix a few issues (#8063) 2022-01-15 23:52:06 +08:00
Sunny Lei 82d04f1924
[DS-6820][feat] Ordinary users can also share resource data source projects with others (#7929)
Admin's entire contents of non-selected users
  Non-admins can only authorize content they create

This closes #6820
2022-01-15 09:08:03 +08:00
springmonster d198a9051e
[Bug-7950] Add unit test for AlertPluginInstanceController (#7952) 2022-01-14 13:52:35 +08:00