One improvement we made in DS vs DCP was to stop hardcoding the job parameters in run.py and just take whatever is there as long as it doesn't have a comment in the name. I think we should do the same thing in the Task Definition- it should just pull the list of imported values*. The only thing that doesn't get us is the ability to override the type in setting up the task definition (ie if we pass an int in the config file but want it to be a str in the task definition), but a) as long as you use the right type in the first place, it should be fine and b) an expert who really did want to change the type could still add that code in run.py . But it will simplify setup for beginners, which I think would be nice.
*= I'm not sure if this is the absolute most elegant way, but it works:
import config; config_vars = [x for x in dir(config) if '__' not in x]
One improvement we made in DS vs DCP was to stop hardcoding the job parameters in
run.pyand just take whatever is there as long as it doesn't have a comment in the name. I think we should do the same thing in the Task Definition- it should just pull the list of imported values*. The only thing that doesn't get us is the ability to override the type in setting up the task definition (ie if we pass an int in the config file but want it to be a str in the task definition), but a) as long as you use the right type in the first place, it should be fine and b) an expert who really did want to change the type could still add that code in run.py . But it will simplify setup for beginners, which I think would be nice.*= I'm not sure if this is the absolute most elegant way, but it works: