cli: change jemalloc defaults

[RaduBerinde]

This commit changes the jemalloc configuration to reduce Cgo total
spikes. This configuration has shown to reduce spikes without any
noticeable impact to performance.

Epic: none
Release note: None

Full experiment results (note the multiple tabs): https://docs.google.com/spreadsheets/d/1Vs9hx0wytkySIvW3DRXI4MXC49b_9Bgkk_IPsoKFH6M/edit?gid=0#gid=0

This is a before/after of sysbench with 3 nodes and 800M rows:

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[RaduBerinde]

Any thoughts on backporting to 25.1? I would be in favor, it seems like a row risk change.

[sumeerbhola]

Great! I'm in favor of backporting.

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @jbowens, @nicktrav, and @RaduBerinde)

---

pkg/cli/start_jemalloc.go line 30 at r1 (raw file):

// // MALLOC_CONF env var. From the jemalloc man page:
// //   The string specified via --with-malloc-conf, the string pointed to by
// //   the global variable malloc_conf, the “name” of the file referenced by

Is this global variable the same as the je_ prefixed one below?

Gemini claimed the following -- wondering whether to believe it:
malloc_conf variable (Deprecated/Rarely Used)

In some (older) versions of jemalloc, there was a global const char * variable called malloc_conf. You might see it referenced in older documentation or code.

This method is generally deprecated and should be avoided in favor of mallctl*

[RaduBerinde]

Yes, it's the same, our build is prepending je_ to all symbols. The latest man page (https://jemalloc.net/jemalloc.3.html, section TUNING) doesn't mention anything about deprecation. Some of the settings are hard or impossible to apply via mallctl after initialization.

[RaduBerinde]

Hm, I see some "Invalid conf pair: muzzy_delay_ms:0" messages in roachtests. I will investigate. Though this is 0 by default in our version anyway so it'd be ok to remove.

[RaduBerinde]

Oh, it's just a typo, it should be decay

[sumeerbhola]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @jbowens, @nicktrav, and @RaduBerinde)

[RaduBerinde]

TFTR!

bors r+

[craig]

Build succeeded:

• local_roachtest
• docker_image_amd64
• unit_tests
• lint
• linux_amd64_build
• linux_arm64_build
• macos_arm64_build
• check_generated_code
• local_roachtest_fips
• macos_amd64_build
• acceptance
• windows_build
• examples_orms
• linux_amd64_fips_build