[GSoC] Add unit tests for tune API

[helenxie-bit]

What this PR does / why we need it:
This PR adds unit tests for tune API.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

• Docs included if any changes are user facing

[helenxie-bit]

/area gsoc

[helenxie-bit]

Ref: #2339

[andreyvelich]

Thank you for doing this @helenxie-bit!

[helenxie-bit]

@andreyvelich The unit test for tune API is ready. I have also fixed some errors in this PR. Please have a look when you have time. Thanks!

[andreyvelich]

Thank you for doing this @helenxie-bit!
/assign @tenzen-y @johnugeorge @deepanker13 @droctothorpe @Electronic-Waste

[helenxie-bit]

@andreyvelich Thank you so much for reviewing this! I’ve replied to all your feedback, and please take a look when you have time.

[Electronic-Waste]

Thanks for your great contribution @helenxie-bit ! I left some suggestions about the name of UTs for your reference

I guess it will be more friendly for other developers to understand these UTs if we could specify the scope of each UTs:)

[helenxie-bit]

Thanks for your great contribution @helenxie-bit ! I left some suggestions about the name of UTs for your reference

I guess it will be more friendly for other developers to understand these UTs if we could specify the scope of each UTs:)

@Electronic-Waste Thank you so much for your suggestions! I have incorporated your advice into the UTs.

[helenxie-bit]

@andreyvelich I’ve added the kubeflow-training dependency to the unit tests, allowing values and parameters to be fetched from there instead of redefining them within the Katib unit tests. You can find the change here:

katib/test/unit/v1beta1/requirements.txt

Line 3 in b24b44f

kubeflow-training[huggingface]==1.8.1

Additionally, I have added verification for the experiment object in two valid flow cases in the following section:

katib/sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py

Lines 587 to 639 in b24b44f

	call_args = mock_create_experiment.call_args
	experiment = call_args[0][0]
	if test_name == "valid flow with custom objective tuning":
	# Verify input_params
	args_content = "".join(
	experiment.spec.trial_template.trial_spec.spec.template.spec.containers[
	0
	].args
	)
	assert "'a': '${trialParameters.a}'" in args_content
	# Verify trial_params
	assert experiment.spec.trial_template.trial_parameters == [
	V1beta1TrialParameterSpec(name="a", reference="a"),
	]
	# Verify experiment_params
	assert experiment.spec.parameters == [
	V1beta1ParameterSpec(
	name="a",
	parameter_type="int",
	feasible_space=V1beta1FeasibleSpace(min="10", max="100"),
	),
	]
	elif test_name == "valid flow with external model tuning":
	# Verify input_params
	args_content = "".join(
	experiment.spec.trial_template.trial_spec.spec.pytorch_replica_specs[
	"Master"
	]
	.template.spec.containers[0]
	.args
	)
	assert (
	'"learning_rate": "${trialParameters.learning_rate}"'
	in args_content
	)
	# Verify trial_params
	assert experiment.spec.trial_template.trial_parameters == [
	V1beta1TrialParameterSpec(
	name="learning_rate", reference="learning_rate"
	),
	]
	# Verify experiment_params
	assert experiment.spec.parameters == [
	V1beta1ParameterSpec(
	name="learning_rate",
	parameter_type="double",
	feasible_space=V1beta1FeasibleSpace(
	min="1e-05", max="5e-05"
	),
	),
	]

Please review when you have time. Thanks!

[helenxie-bit]

@andreyvelich Please review this PR when you have time 👀. I'll adjust the e2e tests for the tune API after this PR is merged, as it fixes some small bugs in the tune API.

[helenxie-bit]

Since @mahdikhashan is currently working on the notebook with instructions for this API, and this PR also includes some bug fixes for the API, @andreyvelich, would it be better to separate the bug fixes into a standalone PR to make the review process more convenient? Please let me know your thoughts.

[andreyvelich]

Since @mahdikhashan is currently working on the notebook with instructions for this API, and this PR also includes some bug fixes for the API, @andreyvelich, would it be better to separate the bug fixes into a standalone PR to make the review process more convenient? Please let me know your thoughts.

Sure, please create separate PR for the bug fixes.

[Electronic-Waste]

Basically LGTM! I left a few comments concerned with objective_metric_name for you.

Thanks for your great contribution! @helenxie-bit

[Electronic-Waste]

It would be better if we also verify the objective_metric_name parameter.

[helenxie-bit]

Thank you for your suggestions! I will add that.

[helenxie-bit]

/rerun-all

[helenxie-bit]

@andreyvelich @Electronic-Waste This PR is ready for review. But I'm not sure why the ci test Publusi Katib Core images/Publish Image (katib-ui, cmd/ui/v1beta1/Dockerfile) kept failing due to network connectivity issue. Do you have any ideas?

[Electronic-Waste]

LGTM! Thanks for your great contribution! As for the CI error, it occurred occasionally. You could rerun it after a while:)

/lgtm
/rerun-all

[andreyvelich]

Thank you for this great contribution @helenxie-bit!
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, Electronic-Waste

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment