Commit Graph

22 Commits (dev)

Author SHA1 Message Date
Jiajie Zhong 59a026d897
[python] Support read config in env variable (#9517)
Add a new method to get config from environment variables
and for now, we have three ways to get config and the
priority is `env-var > custom-config-file > built-in-config-file`.

Environment config setting does not work in CLI, because it will
confuse users when they get config value is `var-env` but value
in the configuration file is `var-in-file`, they may not find
the way how to change it

* Add documentation
* Add it to UPDATING.md

close: #8344
2022-04-15 15:46:44 +08:00
Jiajie Zhong 5289b09817
[python] Fix change exists pd attribute user error (#9140)
* [python] Fix change exists pd attribute user error

* Remove attribute from class ProcessDefinition, cause it should to
  object User instead of object ProcessDefinition.
* Grant project to user if attribute user change for exists ProcessDefinition

close: #8751

* Add py.test conftest.py for package integration
2022-03-24 15:24:02 +08:00
Jiajie Zhong 30746d9762
[python] Add integrated test to python gateway server (#8966)
* [python] Add integrated test to python gateway server

* Build java code and create standalone server image in GA
* Add component start docker in python
* Run example to make sure it work to it

close: #8035

* Fix build docker image working directory

* Fix working directory
2022-03-18 16:24:40 +08:00
JieguangZhou 32a5ccac72
[python] Add custom log for meanful an easier log formatter (#8901)
* Change logging.warning to logger.warning
* Add testing for logging about task code already in process definition

close: #8258
2022-03-15 19:16:33 +08:00
Jiajie Zhong 5c640789c3
[python] refactor yaml file parser (#8701)
* [python] refactor yaml file parser

* Change yaml parser package to ruamel.yaml
* Refactor configuration.py module
* And file.py to write file locally
* Add more tests on it

close: #8593

* Fix UT error

* Remove pypyaml from tests

* Fix file error when param create is False

* Fix error logic
* And tests to avoid regression
2022-03-07 11:42:28 +08:00
Jiajie Zhong ec4ce2b573
[python] Add config mechanism and cli subcommand config (#8585)
* [python] Add config mechanism and cli subcommand config

* Add configuration.py mechanism for more easy change config
  and move some configs to it. It mechanism including
  configuration.py module and default_config.yaml file
* Add `config` for cli subcommand allow users initialize, get,
  set configs

close: #8344

* Change setup.py format
2022-03-01 11:16:44 +08:00
Devosend 027af091c7
[Feature-7346] [Python]Add workflow as code task type spark (#7968)
* add spark task

* fix code format

* add parent class for flink and spark

* modify Engine docstring

* modify docstring of Engine
2022-01-15 23:24:55 +08:00
Devosend 891b5663a8
[python] Fix global params bug (#7971) 2022-01-14 10:41:46 +08:00
Jiajie Zhong b1edce2b11
[python] Add checker pd attr param and fix switch example (#7818)
* [python] Add checker pd attr param and fix switch example

* Correct submit self.param to java gateway
* Fix missing parameter for switch example
* Add mechanism checker before submit to java gateway

close: #7803

* Fix mock get task code and version

* Change judge statement and add comment
2022-01-05 19:59:20 +08:00
Jiajie Zhong b1afd99c9e
[python] Fix task relation wrong hashable function (#7628)
* [python] Fix task relation wrong hashable function

Currently our :class:`TaskRelation` have wrong __hash__
function, would mistake a object as same object. It failed
our dependence and process definition. this patch fix it
by adding `_KEY_ATTR` to overwrite TaskRelation's __eq__
and correct __hash__ func

close: #7627

* Fix code format
2021-12-26 15:44:30 +08:00
Devosend 0c7aa4e2c5
[Feature-7349] [Python]Add workflow as code task type datax (#7437)
* support datax task by python
* add task datax example

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>
2021-12-17 11:12:48 +08:00
Jiajie Zhong 2a7d6b468f
[python] Refactor get object define communicate to gateway (#7272)
* Change class Base `to_dict` to `get_define` for more clearer
* Remove class TaskParams and sub class make code easy to
  understand and avoid task implement cycle dependence
* Remove class ObjectJsonBase in Task to reduce complexity

fix: #7271
2021-12-08 22:17:13 +08:00
Jiajie Zhong d93248acbc
[python] Add custom exception (#6935) 2021-11-23 16:59:38 +08:00
Jiajie Zhong 6e812be14d
[python] Add ut cover process definitions separate mode (#6908)
* [python] Add ut cover process definitions separate mode

* Remove unused print syntax

* Fix test error, cause raw_script not exists anymore
2021-11-19 17:24:19 +08:00
Jiajie Zhong 0dce68edd7
[python] Add task type python http (#6906)
* [python] Add task type python http

* Fix unittest error

* Fix UT error
2021-11-19 16:37:22 +08:00
Jiajie Zhong 7c5c7ec907
[ci][python] Add isort to sort out import (#6871)
* Add isort config file to fix conflict with black
* Add some doc about isort
2021-11-17 11:48:52 +08:00
Jiajie Zhong 54933b33e3
[ci][python] Add coverage check in CI (#6861)
* [ci] Add coverage check in CI

* Coverage add dependent

* Install pydolphinscheduler before run coverage

* Up test coverage to 87% and down threshold to 85%

* Fix code style

* Add doc about coverage
2021-11-17 09:46:40 +08:00
Jiajie Zhong e76cf77040
[python] Add parameter schedule for process definition (#6664)
* [python] Add parameter schedule for process definition

* Rebase and fix some code style

* May schedule work on both string and datetime

* Fix flaky test

* Add comment about freeze time

* Add edge test for schedule_json with None schedule

* Fix test function name

* Fix rebase error
2021-11-13 17:21:40 +08:00
Jiajie Zhong cd8205217a
[python] Fix process definition's user assigned not pass (#6753) 2021-11-13 16:47:08 +08:00
Jiajie Zhong 32d4411469
[python] Add flake8 and black for code style and integrated to GA (#6679)
* [python] Add code style lint for GA

* Change github action name

* Auto change by black

* Fix flake8

* Fix broken link for pyds README.md

* Auto fix by black

* Separate GitHub workflows

* Add Black badge and CI locally in README.md
2021-11-05 15:14:28 +08:00
Jiajie Zhong 1165afbdd1
[python] Add param workgroup to process definition (#6667) 2021-11-04 22:25:45 +08:00
Jiajie Zhong 2e7036529c
Add CI test for python API (#6636)
* Add CI for python API

* Fix task assign without process definition

* Fix ci

* Add header

* Add build badge for pydolphinscheduler

* Add content to README.md

* Remove unnecessary blank line
2021-11-04 11:19:11 +08:00