Commit Graph

41 Commits (dev)

Author SHA1 Message Date
陈家名 8a8b63cd96
[Improve][python] Support create table syntax and custom sql type param (#9673) 2022-04-25 10:17:20 +08:00
Jiajie Zhong 1f48601c75
[python] Add task decorator for python function (#9496)
* [python] Add task decorator for python function

* Add decorator `@task`
* Add a tutorial about it
* Change tutorial doc and combine into traditional docs
  * Add sphinx-inline-tab for better view

* revert not need change

* Correct python function indent

* Correct integration test
2022-04-15 15:50:52 +08:00
Jiajie Zhong 59a026d897
[python] Support read config in env variable (#9517)
Add a new method to get config from environment variables
and for now, we have three ways to get config and the
priority is `env-var > custom-config-file > built-in-config-file`.

Environment config setting does not work in CLI, because it will
confuse users when they get config value is `var-env` but value
in the configuration file is `var-in-file`, they may not find
the way how to change it

* Add documentation
* Add it to UPDATING.md

close: #8344
2022-04-15 15:46:44 +08:00
Jiajie Zhong 3457cee960
[python] Migrate pythonGatewayServer into api server (#9372)
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server

The distribute package size before and after this patch is:

```sh
# before
796M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz

# after
647M   apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
2022-04-07 14:41:15 +08:00
Jiajie Zhong 5289b09817
[python] Fix change exists pd attribute user error (#9140)
* [python] Fix change exists pd attribute user error

* Remove attribute from class ProcessDefinition, cause it should to
  object User instead of object ProcessDefinition.
* Grant project to user if attribute user change for exists ProcessDefinition

close: #8751

* Add py.test conftest.py for package integration
2022-03-24 15:24:02 +08:00
Jiajie Zhong 30746d9762
[python] Add integrated test to python gateway server (#8966)
* [python] Add integrated test to python gateway server

* Build java code and create standalone server image in GA
* Add component start docker in python
* Run example to make sure it work to it

close: #8035

* Fix build docker image working directory

* Fix working directory
2022-03-18 16:24:40 +08:00
JieguangZhou 32a5ccac72
[python] Add custom log for meanful an easier log formatter (#8901)
* Change logging.warning to logger.warning
* Add testing for logging about task code already in process definition

close: #8258
2022-03-15 19:16:33 +08:00
Jiajie Zhong fb772c328d
[python] Change name to_string to __str__ and add __repr__ (#8830)
* Change `yaml_parser.py` method `to_string` to magic method
  `__str__` make it more pythonic.
* Add some tests to magic method `__str__` and `__repr__`
2022-03-12 20:36:51 +08:00
Jiajie Zhong 5c640789c3
[python] refactor yaml file parser (#8701)
* [python] refactor yaml file parser

* Change yaml parser package to ruamel.yaml
* Refactor configuration.py module
* And file.py to write file locally
* Add more tests on it

close: #8593

* Fix UT error

* Remove pypyaml from tests

* Fix file error when param create is False

* Fix error logic
* And tests to avoid regression
2022-03-07 11:42:28 +08:00
Jiajie Zhong ec4ce2b573
[python] Add config mechanism and cli subcommand config (#8585)
* [python] Add config mechanism and cli subcommand config

* Add configuration.py mechanism for more easy change config
  and move some configs to it. It mechanism including
  configuration.py module and default_config.yaml file
* Add `config` for cli subcommand allow users initialize, get,
  set configs

close: #8344

* Change setup.py format
2022-03-01 11:16:44 +08:00
Jiajie Zhong 3025f67d37
[python] Add mechanism cli only with version as subcommand (#8516)
* Add basic cli mechanism with Click, for now just including
  one single subcommand `version`
* Add general and easy test class in tests/testing/cli, and
  test to version
* Add sphinx-click to general cli docs basic on click
2022-02-24 16:05:55 +08:00
Eric Gao 49a1270ef5
[Feature-8222][python] move examples into the scope of source package (#8340) 2022-02-14 15:07:00 +08:00
Jiajie Zhong 35578efd08
[python] Initiate document for pydolphinscheduler (#8005) 2022-01-26 12:37:57 +08:00
Devosend 7aeac75860
[Feature-7348] [Python] Add workflow as code task type mr (#8140) 2022-01-21 09:54:36 +08:00
Devosend 027af091c7
[Feature-7346] [Python]Add workflow as code task type spark (#7968)
* add spark task

* fix code format

* add parent class for flink and spark

* modify Engine docstring

* modify docstring of Engine
2022-01-15 23:24:55 +08:00
Devosend 891b5663a8
[python] Fix global params bug (#7971) 2022-01-14 10:41:46 +08:00
Jiajie Zhong b1edce2b11
[python] Add checker pd attr param and fix switch example (#7818)
* [python] Add checker pd attr param and fix switch example

* Correct submit self.param to java gateway
* Fix missing parameter for switch example
* Add mechanism checker before submit to java gateway

close: #7803

* Fix mock get task code and version

* Change judge statement and add comment
2022-01-05 19:59:20 +08:00
Jiajie Zhong 1417967d9e
[python] Task condition missing two downstream param (#7783)
* [python] Task condition missing two downstream param

We add two downstream tasks to set task condition
success and failed node

close: #7763

* Add getter and setter property condition_resulth in base task
2022-01-05 19:58:44 +08:00
Jiajie Zhong f324b2f884
[python] Add test for examples (#7759)
* Make sure all file with `.py` extension and all file
  end with "_example"
* Make sure all examples have `__doc__`
* Make sure not have duplicate process definition name
* Make sure process definition same as filename

close: #7729
2022-01-04 10:09:38 +08:00
Devosend 081adf4aaa
[Feature-7347] [Python]Add workflow as code task type flink (#7632)
* add Flink task for code as flow

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2022-01-04 09:45:54 +08:00
Jiajie Zhong cc8fbe3e14
[python] Fix task condition set wrong deps (#7650)
After #7505 merged. we could use condition task type
but our dependent set in the wrong direction, all
the condition operators should be upstream of the
current task instead of downstream

fix: #7649
2021-12-27 20:37:40 +08:00
Jiajie Zhong b1afd99c9e
[python] Fix task relation wrong hashable function (#7628)
* [python] Fix task relation wrong hashable function

Currently our :class:`TaskRelation` have wrong __hash__
function, would mistake a object as same object. It failed
our dependence and process definition. this patch fix it
by adding `_KEY_ATTR` to overwrite TaskRelation's __eq__
and correct __hash__ func

close: #7627

* Fix code format
2021-12-26 15:44:30 +08:00
Jiajie Zhong 946a0c7c57
[python] Add task switch (#7531)
* [python] Add task switch

close: #6928

* Fix code style
2021-12-22 11:46:34 +08:00
Jiajie Zhong e23a4848c0
[python] Add task condition (#7505)
* [python] Add task condition

close: #6927

* Add example and set downstream dep
2021-12-22 11:06:45 +08:00
Devosend 0c7aa4e2c5
[Feature-7349] [Python]Add workflow as code task type datax (#7437)
* support datax task by python
* add task datax example

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2021-12-17 11:12:48 +08:00
Jiajie Zhong 6f93ebf3ba
[python] Add task dependent (#7405)
fix: #6926
2021-12-16 09:58:50 +08:00
Jiajie Zhong 4988004150
[python] Fix task procedure define error (#7407)
Cause procedure parameter name need `method` instead of `sql`
we change it's name. And remove parameter `sql` from parent
class `Database`

closes: #7386
2021-12-15 10:19:32 +08:00
Jiajie Zhong 1948030151
[python] Add task base database and procedure (#7279)
We add a new task procedure, and add parent class database
for both sql task and procedure task

fix: #6929
2021-12-13 19:54:24 +08:00
Jiajie Zhong 2a7d6b468f
[python] Refactor get object define communicate to gateway (#7272)
* Change class Base `to_dict` to `get_define` for more clearer
* Remove class TaskParams and sub class make code easy to
  understand and avoid task implement cycle dependence
* Remove class ObjectJsonBase in Task to reduce complexity

fix: #7271
2021-12-08 22:17:13 +08:00
Devosend f480b8adf8
[Feature-6930] [Python]Add workflow as code task type sub_process (#7022)
* [python] add subProcess task

* refactor python gateway server and task for subprocess

* add function comment for getProcessDefinition

* change process_definition usage of Subprocess task
2021-12-03 10:26:45 +08:00
Jiajie Zhong d93248acbc
[python] Add custom exception (#6935) 2021-11-23 16:59:38 +08:00
Jiajie Zhong 41e8836c91
[python] Add task sql (#6968)
* [python] Add task sql

* Add java gateway function doc
2021-11-23 16:58:00 +08:00
Jiajie Zhong 6e812be14d
[python] Add ut cover process definitions separate mode (#6908)
* [python] Add ut cover process definitions separate mode

* Remove unused print syntax

* Fix test error, cause raw_script not exists anymore
2021-11-19 17:24:19 +08:00
Jiajie Zhong 0dce68edd7
[python] Add task type python http (#6906)
* [python] Add task type python http

* Fix unittest error

* Fix UT error
2021-11-19 16:37:22 +08:00
Jiajie Zhong 7c5c7ec907
[ci][python] Add isort to sort out import (#6871)
* Add isort config file to fix conflict with black
* Add some doc about isort
2021-11-17 11:48:52 +08:00
Jiajie Zhong 54933b33e3
[ci][python] Add coverage check in CI (#6861)
* [ci] Add coverage check in CI

* Coverage add dependent

* Install pydolphinscheduler before run coverage

* Up test coverage to 87% and down threshold to 85%

* Fix code style

* Add doc about coverage
2021-11-17 09:46:40 +08:00
Jiajie Zhong e76cf77040
[python] Add parameter schedule for process definition (#6664)
* [python] Add parameter schedule for process definition

* Rebase and fix some code style

* May schedule work on both string and datetime

* Fix flaky test

* Add comment about freeze time

* Add edge test for schedule_json with None schedule

* Fix test function name

* Fix rebase error
2021-11-13 17:21:40 +08:00
Jiajie Zhong cd8205217a
[python] Fix process definition's user assigned not pass (#6753) 2021-11-13 16:47:08 +08:00
Jiajie Zhong 32d4411469
[python] Add flake8 and black for code style and integrated to GA (#6679)
* [python] Add code style lint for GA

* Change github action name

* Auto change by black

* Fix flake8

* Fix broken link for pyds README.md

* Auto fix by black

* Separate GitHub workflows

* Add Black badge and CI locally in README.md
2021-11-05 15:14:28 +08:00
Jiajie Zhong 1165afbdd1
[python] Add param workgroup to process definition (#6667) 2021-11-04 22:25:45 +08:00
Jiajie Zhong 2e7036529c
Add CI test for python API (#6636)
* Add CI for python API

* Fix task assign without process definition

* Fix ci

* Add header

* Add build badge for pydolphinscheduler

* Add content to README.md

* Remove unnecessary blank line
2021-11-04 11:19:11 +08:00