Commit Graph

43 Commits (de50f43de63e42488cb38169cd56fba77dd24fc6)

Author SHA1 Message Date
Jiajie Zhong de50f43de6
[common] Make dolphinscheduler_env.sh work when start server (#9726)
* [common] Make dolphinscheduler_env.sh work

* Change dist tarball `dolphinscheduler_env.sh` location
  from `bin/` to `conf/`, which users could finish their
  change configuration operation in one single directory.
  and we only need to add `$DOLPHINSCHEDULER_HOME/conf`
  when we start our sever instead of adding both
  `$DOLPHINSCHEDULER_HOME/conf` and `$DOLPHINSCHEDULER_HOME/bin`
* Change the `start.sh`'s path of `dolphinscheduler_env.sh`
* Change the setting order of `dolphinscheduler_env.sh`
* `bin/env/dolphinscheduler_env.sh` will overwrite the `<server>/conf/dolphinscheduler_env.sh`
when start the server using `bin/dolphinsceduler_daemon.sh` or `bin/install.sh`
* Change the related docs
2022-04-25 15:35:43 +08:00
songjianet 0d0674df1f
[Fix][UI Next][V1.0.0-Alpha] Fix the cannot update tenant individually on user info site. (#9237) 2022-03-28 17:55:15 +08:00
nobolity 0e3cafec1d
[Feature-8612][RESOURCE] extend s3 to the storage of ds (#8637)
* feat(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config

* fix(resource  manager): update the dependency

* fix(resource  manager): extend s3 to the storage of ds

fix the constant of hadooputils

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager):

merge  the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl

* fix(resource  manager): extend s3 to the storage of ds

merge the resourceServiceTest

* fix(resource  manager): test  cancel the test method

createTenant verifyTenant

* fix(resource  manager): merge the code  follow the check-result of sonar

* fix(resource  manager): extend s3 to the storage of ds

fit the spell question

* fix(resource  manager): extend s3 to the storage of ds

revert the common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the storageConfig with None

* fix(resource  manager): extend s3 to the storage of ds

fix the judge of resourceType

* fix(resource  manager): extend s3 to the storage of ds

undo the compile-mysql

* fix(resource  manager): extend s3 to the storage of ds

delete hadoop aws

* fix(resource  manager): extend s3 to the storage of ds

update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties

* fix(resource  manager): extend s3 to the storage of ds

update the aws-region

* fix(resource  manager): extend s3 to the storage of ds

fix the storageconfig init

* fix(resource  manager): update e2e docker-compose

update e2e docker-compose

* fix(resource  manager): extend s3 to the storage of ds

revent the e2e common.proprites

print the resource type in propertyUtil

* fix(resource  manager): extend s3 to the storage of ds
1.println the properties

* fix(resource  manager): println the s3 info

* fix(resource  manager): extend s3 to the storage of ds

delete the info  and upgrade the s3 info to e2e

* fix(resource  manager): extend s3 to the storage of ds

add the bucket init

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): extend s3 to the storage of ds

upgrade the s3 endpoint

* fix(resource  manager): withPathStyleAccessEnabled(true)

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* fix(resource  manager): upgrade the  s3client builder

* fix(resource  manager): correct  the s3 point to s3client

* fix(resource  manager): update the constant BUCKET_NAME

* fix(resource  manager): e2e  s3 endpoint -> s3:9000

* fix(resource  manager): extend s3 to the storage of ds

1.fix some spell question
2.delete the import *

* style(resource  manager): add info to createBucket

* style(resource  manager): debug the log

* ci(resource  manager): test

test s3

* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql

* fix(resource  manager): update the h2 sql

* fix(resource  manager): solve to delete the tenant

* style(resource  manager): merge the style end delete the unuse s3 config

* fix(resource  manager): extend s3 to the storage of ds

UPDATE the rename resources when s3

* fix(resource  manager): extend s3 to the storage of ds

1.fix the code style of QuartzImpl

* fix(resource  manager): extend s3 to the storage of ds

1.impoort restore_type to CommonUtils

* fix(resource  manager): update the work thread

* fix(resource  manager): update  the baseTaskProcessor

* fix(resource  manager): upgrade dolphinscheduler-standalone-server.xml

* fix(resource  manager): add  user Info to dolphinscheduler_h2.sql

* fix(resource  manager): merge  the resourceType to NONE

* style(upgrade the log level to info):

* fix(resource  manager): sysnc the h2.sql

* fix(resource  manager): update the merge the user tenant

* fix(resource  manager): merge the resourcesServiceImpl

* fix(resource  manager):

when the storage is s3 ,that the directory can't be renamed

* fix(resource  manager): in s3 ,the directory cannot be renamed

* fix(resource  manager): delete the deleteRenameDirectory in E2E

* fix(resource  manager): check the style and  recoverd the test

* fix(resource  manager): delete the log.print(LoginUser)
2022-03-25 13:28:13 +08:00
labbomb da2e85e9ec
[BUG][UI Next]fix the problem about Batch delete not working (#9080)
* fix the problem about Batch delete not working while click the border of the button

* try to fix e2e

Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
2022-03-24 14:43:02 +08:00
xiangzihao 197accc601
[Feature][e2e] Suggest e2e test adapt M1 chip (#9077) 2022-03-22 14:11:50 +08:00
xiangzihao e0e7701341
[E2E] [CI] Fix flaky e2e test cause by #9036 (#9053) 2022-03-21 17:30:33 +08:00
Devosend d91711b322
[Fix][UI Next][V1.0.0-Alpha] Menu error (#9036)
* modify the sidebar menu control mode

* modify side menu

* store last path

* fix security auth error

* fix resource e2e

* fix resource manage e2e bug
2022-03-21 15:19:27 +08:00
kezhenxu94 7cf79da5d9
Reduce the size of distribution tar (#9019) 2022-03-20 16:42:06 +08:00
xiangzihao 82394ba81d
[Feature-9021][E2E] Recover sub_process e2e test in ui-next (#9023) 2022-03-20 13:11:48 +08:00
xiangzihao eefe7896c8
[Feature-8850][E2E] Restore project center e2e test cases in ui-next (#8909) 2022-03-15 22:07:30 +08:00
Kerwin a1bef43981
[Feature-8579][Task] Task back-end plugin optimization, new plugins only need to modify the plugin's own module (#8584)
* Task plug-in optimization, task-related classes are migrated to the task plugin module

* remove tasktype enum class.

* fix code style.

* fix code style

* fix code style

* fix ProcessServiceTest.testUpdateResourceInfo test class

* fix code style

* fix test task plugin manager check parameters null

* fix test ut

* update aws license

* change TaskTimeoutStrategy class
2022-03-15 15:16:36 +08:00
xiangzihao 2335453482
[Feature-8844][E2E] Restore datasource center e2e test cases in ui-next (#8845) 2022-03-13 12:33:19 +08:00
xiangzihao 49b1a073dd
[Feature-8823][E2E] Restore resource center e2e test cases in ui-next (#8840) 2022-03-12 20:49:46 +08:00
xiangzihao 20dd0c12ce
[E2E] Restore security center e2e test cases in ui-next (#8815) 2022-03-10 21:49:04 +08:00
kezhenxu94 d2a360480a
[CI][UI NEXT] Package new UI into release tar and remove old UI from it (#8670)
* [CI][UI NEXT] Package new UI into release tar and remove old UI from it

* Update e2e.yml
2022-03-07 11:20:24 +08:00
qianli2022 151100562f
[Feature][api] Task can run on kubernetes - k8s client and namespace #8247 (#8252)
* add db operator for namespace

* add k8s client

* add k8s controller

* add namespace ui

* add licenses

* add kubernetes model known-dependencies

* add new licenses

* fix SonarCloud Code Analysis bug

* remove license for standard protocol

* fix variable MemoryStr to memoryStr

Co-authored-by: qianl4 <qianl4@cicso.com>
2022-02-15 10:21:15 +08:00
wind fc9a31f813
[E2E] add switch task case (#8180)
* fix switch bug

* add switch e2e test

* add switch workflow

* pre task option

* selectPreTask id

* update

* switch task e2e

* naming optimization

Co-authored-by: caishunfeng <534328519@qq.com>
2022-02-10 10:00:50 +08:00
QuakeWang fd38ef129b
[E2E] Add the update test for tenant (#8282) 2022-02-08 09:25:00 +08:00
xiangzihao 43bd640aad
[E2E] Add hive datasource e2e test and fix s3 docker image latest tag (#8276) 2022-02-06 14:22:38 +08:00
QuakeWang 63f7efa993
[E2E] Add new test of token management (#8161) 2022-01-30 12:34:50 +08:00
xiangzihao 4b58df9aa8
[E2E] add function manage test case (#8250) 2022-01-30 09:01:19 +08:00
xiangzihao 1ce5b5d82d
fix_flaky_worker_group_e2e_test (#8237) 2022-01-28 14:15:33 +08:00
xiangzihao 6094927dda
[E2E] Fix flaky udf manage e2e test (#8232)
* fix_file_manage_e2e_error

* fix error

* fix error

* fix error
2022-01-28 11:49:55 +08:00
wind 756ea1181e
[Improvement][E2E] use testcontainers host (#8224) 2022-01-27 19:15:05 +08:00
xiangzihao d6690fa89c
[E2E] add udf manage test (#8203) 2022-01-27 16:17:32 +08:00
xiangzihao 7b22c16a85
[E2E]Fix file manage e2e error (#8214) 2022-01-27 14:59:22 +08:00
kezhenxu94 db953a229c
[E2E] Clean up the E2E with Java API to avoid platform sensitive (#8197) 2022-01-26 14:33:11 +08:00
xiangzihao 30b8b4a93e
[E2E][CI] Add renaming file management E2E tests (#8160) 2022-01-26 09:45:21 +08:00
xiangzihao e7f480bdca
[E2E][CI] Add DataSource E2E test (#8085) 2022-01-19 17:29:02 +08:00
xiangzihao b705a08794
[Feature-7993][E2E] Replace HDFS image with MinIO S3 image (#8066) 2022-01-17 19:32:47 +08:00
Jiajie Zhong a061cf136c
[e2e] Add new test of security environment (#8048)
* [e2e] Add new test of security environment

ref: #7281

* Fix frontend bug

* Fix order of UT

* Fix delete and edit bottom class

* Try to fix

* Try to fix
2022-01-15 09:13:04 +08:00
xiangzihao 5be8e0531c
[E2E] [Resource] add file manage case (#7906) 2022-01-12 14:29:03 +08:00
wind 3edc58cbdd
[E2E] Add queue e2e test (#7941) 2022-01-12 10:31:17 +08:00
janeHe13 1ff339f37b
[E2E] Modify the tenant delete of the e2e case (#7921)
* Update UserE2ETest.java
2022-01-10 17:06:32 +08:00
janeHe13 0b38d0e826
[E2E]add worker group manage e2e case (#7879)
* add worker group manage e2e case

* add worker group manage e2e case

* Update dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/WorkerGroupE2ETest.java

Co-authored-by: janeHe13 <438044805@qq.com>
Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-01-07 21:24:03 +08:00
Jiajie Zhong 079adc9eca
[ci] Fix flaky test in e2e user (#7847)
* [ci] Fix flaky test in e2e user

fix: #7656

* Change class name

* Change get delete bottom function

* Correct function
2022-01-06 17:29:06 +08:00
kezhenxu94 269a62418f
Add E2E case for user manage (#7652) 2021-12-28 14:02:06 +08:00
kezhenxu94 03a71297dd
Unify the element selector rules for E2E (#7409)
* Unify the element selector rules for E2E

* Fix typo in docker swarm
2021-12-22 14:53:39 +08:00
kezhenxu94 67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
kezhenxu94 37ba1eb5ad
Add E2E tests for some core features (#7025) 2021-11-30 13:07:55 +08:00
kezhenxu94 ef0cb46661
Fix shell tools for database schema cannot run (#7003) 2021-11-26 11:10:38 +08:00
kezhenxu94 4114cb07f6
Add actuator and enable metrics, add Grafana configs (#6840) 2021-11-14 20:22:13 +08:00
kezhenxu94 251255009a
Add end-to-end test framework and some basic cases (#6419) 2021-10-20 21:14:26 +08:00