Add CP_OPT

[ntjohnson1]

Goal: Align implementation with GCP_OPT while retaining the cp_opt top level interface as faithfully as possible.

Components:

• cp_opt.m
• ktensor/fg.m
• tt_opt_lbfgsb.m
• tt_opt_adam.m
• cp_opt_doc.m

Won't do unless needed/requested as follow up:

• cp_opt_legacy.m
• tt_opt_lbfgs.m, Poblano isn't available in python, scipy has a standard BFGS solve but not LBFGS. With the other implementations adding a new optimizer shouldn't be too much of a burden
• tt_opt_compact.m The solver from this paper is publicly available but uses pytorch. That's a pretty heavy include for our project (also it doesn't look like they publish the solver to pypi)
• tt_opt_fmincon.m/ tt_opt_fminunc.m Both of these seem to use matlab optimization tools that run a variety of heuristics before dispatching to a specific solver. We have an example of lbfgsb from scipy so if there are specific solvers desired we could add those. But would be hard to come up with an apples to apples comparison.

[dmdunla]

I think that scipy’s L-BFGS-B (via minimize) without constraints reduces to L-BFGS. Will that work to support the tt_opt_lbfgs.m use case you called out for cp_opt?

…

________________________________ From: Nick ***@***.***> Sent: Saturday, November 23, 2024 10:15 AM To: sandialabs/pyttb ***@***.***> Cc: Subscribed ***@***.***> Subject: [EXTERNAL] [sandialabs/pyttb] Add CP_OPT (Issue #351) Goal: Align implementation with GCP_OPT while retaining the cp_opt top level interface as faithfully as possible. Components: [ ] cp_opt.m [ ] ktensor/fg.m [ ] tt_opt_lbfgsb.m [ ] tt_opt_fmincon.m [ ] tt_opt_fminunc.m [ ] tt_opt_compact.m [ ] tt_opt_adam.m Won't do unless needed/requested as follow up: * cp_opt_legacy.m * tt_opt_lbfgs.m, Poblano isn't available in python, scipy has a standard BFGS solve but not LBFGS. With the other implementations adding a new optimizer shouldn't be too much of a burden — Reply to this email directly, view it on GitHub<#351>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHIY2IVG4U3YLTJ27INH45L2CCZ4LAVCNFSM6AAAAABSLIQZ4KVHI2DSMVQWIX3LMV43ASLTON2WKOZSGY4DMMRYGM4TMNA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ntjohnson1]

I can take a look at the end (or potentially do it as a follow up).

[ntjohnson1]

I timed out for this afternoon but here is my development branch for this: https://github.com/ntjohnson1/pyttb/tree/nick/cp_opt

I have most of cp_opt, function/gradient handles for the specific cp opt problem, and lbfgsb implemented. Still need to improve doc strings, and tests to confirm things are functionally correct (right now I have a simple test to confirm when the answer is provided for init things word). I basically followed the same internal structure for the solvers in gcp_opt, however the cp_opt components operate on actual tensors rather than just blocks of data so I had to slightly tweak some interfaces. No rush, but feel free to share if you have any thoughts on the layout/alignment with gcp_opt.

I also updated the description for some of the other optimizers python had used. My preference would be to implement a subset of the optimizers if the primary motivation is the textbook rather than real world usage.

[dmdunla]

SG. I think the subset of optimizers you landed on looks good to me. I do not think that we need to replicate Poblano or the Optimization Toolbox for MATLAB routines as a whole as a start. I think that main point is to have an all-at-once method for CP-OPT with reasonable optimizer support first, then flesh this out with other methods as needed.

Have you thought about the extension of CP-OPT to CP-WOPT, in terms of the interface? Those were developed separately in MATLAB, but there may be an opportunity to consider fusion of these in terms of the API, as Stephen Becker's L-BFGS-B tool used by MATLAB does.

[ntjohnson1]

No. I basically got to this point and paused. I can start looking at WOPT before cleaning up/fine tuning this to make sure we can support that with a similar interface as well.

[tgkolda]

I have cleaning up the MATLAB codes on my to-do list. Once I do that, maybe the structure will be clearer here.

[ntjohnson1]

I have cleaning up the MATLAB codes on my to-do list. Once I do that, maybe the structure will be clearer here.

I think right now MATLAB, pyttb, and genten all have slightly different formats for GCP. Here we were aiming to align the pyttb GCP format with CP-OPT and CP-WOPT. Feel free to take a look at our setup, or we can chat if that's helpful. The top level call to gcp/cpopt/cpwopt should remain the same but the organization of the code is different. Here we tried to align more with a pytorch/sklearn/machine learning based organization with the optimizers etc since that seems to be most familiar for python users.