Add Support HTTP Multiplexing

[AiraYumi]

Enables HTTP/2 Multiplexing.
If this doesn't work, it will use HTTP pipelining.

(Need Help)
I want to be able to enable and disable HTTP/2 Multiplexing using the debug options, but where should I change it? maybe solve

[marchcat]

Thank you for the contribution!

(Need Help)
I want to be able to enable and disable HTTP/2 Multiplexing using the debug options, but where should I change it?

To achieve this, add a setting to the indra/newview/app_settings/settings.xml file as follows:

    <key>UseHTTP2Multiplexing</key>
    <map>
      <key>Comment</key>
      <string>Enable HTTP/2 Multiplexing</string>
      <key>Persist</key>
      <integer>1</integer>
      <key>Type</key>
      <string>Boolean</string>
      <key>Value</key>
      <integer>0</integer>
    </map>

Then, access it from the viewer code like this:

bool use_http2_multiplexing = gSavedSettings.getBOOL("UseHTTP2Multiplexing");

[akleshchev]

I want to be able to enable and disable HTTP/2 Multiplexing using the debug options, but where should I change it?

I doubt that all services will pick up http/2 in one go, so it's probably beneficial to do it on a case by case bassis viewer side, like HttpOptions (or header?).
In such case calls will also be in newview, so you will be able to use settings to do something like mHttpOptions->setUseHttp2(gSavedSettings.getBOOL("UseHTTP2Multiplexing"))

To achieve this, add a setting to the indra/newview/app_settings/settings.xml file as follows:

I doubt that it's accessible from curl's files.

P.S. using gSavedSettings.getBOOL() directly for a code like this is suboptimal, it probably should be LLCachedControl inited from gSavedSettings...

[AiraYumi]

I'm currently working on it, and at least I added gSavedSettings to "indra/llcorehttp/_httplibcurl.cpp", but it didn't build.
I'll try adding it to HttpOptions.

[akleshchev]

You can also remake mPipelined into a enum with options like no|pipelining|multiplexing. It already loads from gSavedSettings.getBOOL(http_pipelining), so it will just have to become getS32(http_pipelining).

[AiraYumi]

I don't understand how to use HttpOptions in _httplibcurl.cpp.
I would like to toggle an option for check_curl_multi_setopt.

[monty-linden]

Per-request and per-multi-handle options are kept well-apart in llcorehttp. So curl_multi_setopt values are controlled by options sent in via httppolicyclass. Per-request options in HttpOptions find their way into _httpoprequest where global, per-policy-class, and per-request options are baked into curl_easy_setopt calls in prepareRequest() (mostly).

It sounds like you want to extend the policy class definition. But maybe describe what libcurl options you want to change and that will shape an answer. There's also the README.Linden file which hasn't been maintained but has useful tutorial information.

[AiraYumi]

It is now possible to switch between CURLPIPE_MULTIPLEX or CURLPIPE_HTTP1 depending on the value registered in httppolicyclass.
The default is 0, so it works the same as before the change.

--- Here's what I think ---
Setting UseHTTP2Multiplexing to 1 will use HTTP/2 Multiplexing, and setting it to 0 will use HTTP Pileline.
Since they operate mutually exclusive, enabling HTTP/2 will disable HTTP Pileline.

[AiraYumi]

I think it should work, let me know if you have any problems.

[akleshchev]

Personally I would have merged "HttpPipelining" and "UseHTTP2Multiplexing" into some kind of 'mode switch' since they are sort of exclusive.

[monty-linden]

One very large problem I have with these http/2 PRs is that this code, while disabled by default, cannot even be tested experimentally anywhere. Linden has a lot of infrastructure changes to make. (https: support, http/2 support, origin server changes, multiple grids, CDN configuration, new certs and cipher negotiation) So this is code that has never been tested and cannot simply be picked up and developed without that work. I approve of the spirit but this may not be the way to start.

Given the amount of infrastructure that must change and the need for testing around the world, this is work that should probably be packaged in a project viewer and not targeted at 'develop'.

I was surprised when these PRs appeared. Is there some particular need driving the interest in HTTP/2? We do have a project outlined but other work must happen first. And now I am curious if there is an important need for http/2 now....

[monty-linden]

Is this debug logging left in? Should it not use the supplied LOG_* tag values above?

[monty-linden]

The nesting conditionals are probably wrong, in practice. If mMultiplexing is on but mPipelining is off, multiplexing ends up disallowed. Probably not intended. One of those areas we'll have to sort out. If the 'mPipelining' concept is changed to concurrency, this is more nearly correct.

[monty-linden]

More explanation about this API. The 'public' API presented here is not meant to be a facade for libcurl. At the time this was written and also possibly for future purposes, libcurl might be replaced by another library. (For Sansar, we had our own low-level libcurl-like library which could have slotted in.) The 'public' API avoided using libcurl definitions.

Instead, this API is meant to describe how the viewer wants to use HTTP requests and how HTTP traffic should be managed and shaped. The intent is to have this description be as minimal as possible. If a concept isn't used by the viewer, it won't appear here.

Internal APIs, such as _httplibcurl.*, are the translation layer between the public API and whatever is going on below. An adapter layer. There's only one concrete adapter at this point but that is okay.

And so we have the comment above:

When PIPELINING_DEPTH is 2 or more, libcurl performs

That is still correct. The public API describes an abstract usage and a pipelining depth of 2 or more enables pipelining. And so the code above in _httplibcurl is no longer correct.

What should this API be in the http/2 world? Well, we don't know yet. We haven't stood up the infrastructure to support it. But we would probably want to modify llcorehttp differently. PO_PIPELINING_DEPTH might instead be changed to PO_REQUEST_CONCURRENCY covering both the http/1 (pipelining) and http/2 (multiplexing) cases with added explicit enables/disables for pipelining and http/2. However, we may find that we need independent http/2 parameters for the policy classes.

(There's another weak reason to keep existing API definitions as-is: we use a copy of this code in server-side binaries and subtle changes may not be caught. This isn't meant to stop changes and progress in the viewer. But when all other considerations are equal, keeping an API spec has a benefit to us.)

[AiraYumi]

The reason we decided to introduce the concept of HTTP/2 to the viewer was the end of support for OpenSSL.

One of the obstacles to updating OpenSSL is updating Curl.
One of the obstacles to updating Curl is the HTTP Pipeline.
When Curl is updated, the HTTP Pipeline becomes unavailable, which means performance is reduced.
For this reason, we feel that the HTTP Pipeline technology is a major obstacle to updating Curl and OpenSSL.

We believe that if HTTP/2 is successfully introduced, one of the obstacles to updating OpenSSL will be removed.
More than 10 years have passed since the concept of the HTTP Pipeline was introduced to the viewer.

During this time, HTTP has been updated to HTTP/3, but the viewer continues to rely on HTTP/1.1...

[AiraYumi]

The issues pointed out that did not work as intended have been fixed.

I now understand that this content is likely to be something that should be put into the project viewer.
I certainly understand the argument that things that can't be tested shouldn't be put into develop.
If some of the CDNs in the beta SIM could support HTTP/2, we'd be able to test it...

[monty-linden]

Yes, there is quite a bit of work that must happen on our side. For now, I recommend making this a 'draft pull request' so that it isn't lost. It may be useful to start here in the future.

I do have an http/2 project outline. I will put this in a better place later. But I can share it here for reference:

Milestone 1 - Testbed

• Build and run viewer with special instrumentation to recover the asset fetch profile for interesting sessions. These are the mix of asset fetch and HTTP range options one will expect on entry to dense regions.
• PoC changes to llcorehttp to support HTTP/2 in the crudest fashion. Used to get an initial benchmark with the above profile. Identifies:
  • Performance gap introduced by https:
  • API changes needed in the library to support dual configuration (both HTTP/1 and HTTP/2).
  • Excessive connection concurrency and head-of-line issues as connections transition to /2.

Milestone 2 - Infrastructure

• Initial asset system and CDN work:
  • Cert creation and distribution.
  • HTTP/2 configuration as needed.
  • CDN origin server modifications.
• Simulator:
  • New and/or modified Capabilities to support HTTP/2 endpoints without impacting existing legacy client code.

Milestone 3 - Internal Viewer

• Llcorehttp modified according to design fallout from above.
• Additional library support updated as needed.
• Viewer that can be used internally by others with some inconveniences.
• TPVs: begin communications

Milestone 4 - Project Viewer

• Llcorehttp and libraries take final form.
• Viewer uses new infrastructure, presents new configuration options, etc.
• Viewer gets any final polishing needed including additional diagnostics for connection problems, etc.
• Publish and share.

Milestone 5 - Final Pass

• Expect additional tuning and control needs after this gets some exposure in the field. Users with particularly bad connectivity or wonky networking equipment may need special consideration.