Commit Graph

76 Commits (dev)

Author SHA1 Message Date
Devosend 0c7aa4e2c5
[Feature-7349] [Python]Add workflow as code task type datax (#7437)
* support datax task by python
* add task datax example

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2021-12-17 11:12:48 +08:00
Jiajie Zhong 6f93ebf3ba
[python] Add task dependent (#7405)
fix: #6926
2021-12-16 09:58:50 +08:00
Jiajie Zhong 4988004150
[python] Fix task procedure define error (#7407)
Cause procedure parameter name need `method` instead of `sql`
we change it's name. And remove parameter `sql` from parent
class `Database`

closes: #7386
2021-12-15 10:19:32 +08:00
Jiajie Zhong 1948030151
[python] Add task base database and procedure (#7279)
We add a new task procedure, and add parent class database
for both sql task and procedure task

fix: #6929
2021-12-13 19:54:24 +08:00
kezhenxu94 67cc260d52
Split the components into individual package, refactor configurations to use Spring properties (#7160)
* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.

* Split the components into individual package

A follow-up PR will be made to build dedicated Docker images for each
component, so that every component Docker image has minimal jars, which
is easy to maintain and good for security fixes.
2021-12-10 11:28:53 +08:00
Jiajie Zhong 2a7d6b468f
[python] Refactor get object define communicate to gateway (#7272)
* Change class Base `to_dict` to `get_define` for more clearer
* Remove class TaskParams and sub class make code easy to
  understand and avoid task implement cycle dependence
* Remove class ObjectJsonBase in Task to reduce complexity

fix: #7271
2021-12-08 22:17:13 +08:00
Devosend f480b8adf8
[Feature-6930] [Python]Add workflow as code task type sub_process (#7022)
* [python] add subProcess task

* refactor python gateway server and task for subprocess

* add function comment for getProcessDefinition

* change process_definition usage of Subprocess task
2021-12-03 10:26:45 +08:00
JinYong Li 5a04b8d49a
update schdule api (#6977) 2021-11-25 10:22:41 +08:00
Jiajie Zhong d93248acbc
[python] Add custom exception (#6935) 2021-11-23 16:59:38 +08:00
Jiajie Zhong 41e8836c91
[python] Add task sql (#6968)
* [python] Add task sql

* Add java gateway function doc
2021-11-23 16:58:00 +08:00
Jiajie Zhong 6e812be14d
[python] Add ut cover process definitions separate mode (#6908)
* [python] Add ut cover process definitions separate mode

* Remove unused print syntax

* Fix test error, cause raw_script not exists anymore
2021-11-19 17:24:19 +08:00
Jiajie Zhong 0dce68edd7
[python] Add task type python http (#6906)
* [python] Add task type python http

* Fix unittest error

* Fix UT error
2021-11-19 16:37:22 +08:00
JinYong Li 4bec792160
[Fix][Common] rewrite code generate,fix bit shift (#6914)
* rewrite code generate,fix bit shift

* fix ut

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* add algorithm from licenses file

* fix ut
2021-11-19 16:04:05 +08:00
Jiajie Zhong 7c5c7ec907
[ci][python] Add isort to sort out import (#6871)
* Add isort config file to fix conflict with black
* Add some doc about isort
2021-11-17 11:48:52 +08:00
Jiajie Zhong 54933b33e3
[ci][python] Add coverage check in CI (#6861)
* [ci] Add coverage check in CI

* Coverage add dependent

* Install pydolphinscheduler before run coverage

* Up test coverage to 87% and down threshold to 85%

* Fix code style

* Add doc about coverage
2021-11-17 09:46:40 +08:00
mask a998415afe
[Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490) (#6828)
* [Feature][datasource] Change DataSource Connection Pool from Druid to HiKariCP (#6490)

* fix pom dep scope

* filter alert

* filter alert

* filter alert

* fix checkstyle

* remove hikaricp version

* update pom

* add properties

* add properties

* rename properties

* fix e2e fail

* rename properties
2021-11-14 13:02:58 +08:00
Jiajie Zhong e76cf77040
[python] Add parameter schedule for process definition (#6664)
* [python] Add parameter schedule for process definition

* Rebase and fix some code style

* May schedule work on both string and datetime

* Fix flaky test

* Add comment about freeze time

* Add edge test for schedule_json with None schedule

* Fix test function name

* Fix rebase error
2021-11-13 17:21:40 +08:00
Jiajie Zhong cd8205217a
[python] Fix process definition's user assigned not pass (#6753) 2021-11-13 16:47:08 +08:00
wangxj3 dc8b87e473
[Feature-#6268][server-master] Serial execte process (#6185)
* add serial processInstance

* fix bug

* add test

* fix code style

* fix style code

* add sql

* fix sql error

* add api

* add test

* fix code style

* modify api

* delete column , fix mapper

* fix unimport

* fix test

* fix bug of missing param for Python

* fix code style

* fix test

* fix test

Co-authored-by: wangxj <wangxj31>
2021-11-05 17:25:45 +08:00
Jiajie Zhong 32d4411469
[python] Add flake8 and black for code style and integrated to GA (#6679)
* [python] Add code style lint for GA

* Change github action name

* Auto change by black

* Fix flake8

* Fix broken link for pyds README.md

* Auto fix by black

* Separate GitHub workflows

* Add Black badge and CI locally in README.md
2021-11-05 15:14:28 +08:00
Jiajie Zhong 1165afbdd1
[python] Add param workgroup to process definition (#6667) 2021-11-04 22:25:45 +08:00
Jiajie Zhong 9fb6d6ba9f
Fix f-strings variable error (#6677) 2021-11-04 16:15:16 +08:00
Jiajie Zhong 2e7036529c
Add CI test for python API (#6636)
* Add CI for python API

* Fix task assign without process definition

* Fix ci

* Add header

* Add build badge for pydolphinscheduler

* Add content to README.md

* Remove unnecessary blank line
2021-11-04 11:19:11 +08:00
Jiajie Zhong 9060f8b462
Fix process definitions exists check error (#6668)
* Fix process definitions exists check error

* Add error log while verify status neither success or exists
2021-11-04 10:17:34 +08:00
Jiajie Zhong ef9de9de3b
Add example about how to bulk create workflow (#6659) 2021-11-03 11:23:09 +08:00
Jiajie Zhong dd6ed36f65
Add Python API implementation of workflows-as-code (#6269)
* Init DS python SDK pydolphinscheduler: python code definition

* Doc first

* Add quick start and developer doc

* Java documentation change

* Add LICENSE-py4j.txt

* Add py4j to release-docs/LICENSE

* Move dependency version to parent pom

* Remove outdated code

* Add tenant parameter to tutorial
2021-10-31 20:35:46 +08:00