Support different N3 local IP Address in Access and Core

[pespin]

Is your feature request related to a problem? Please describe.

Some users require running an UPF to forward GTPv1U traffic from network A to network B and viceversa, where those 2 networks are not reachable to each other, only through the UPF host.
This happens for instance when cellular operators don't control/own the entirety of their networks, e.g. a network which runs their NAT under a satellite link with private addresses (eg VPN) which end up at a single host.

As a result, the UPF requires managing at least 2 separate local IP addresses for GTP-U traffic: One (usually private) IP address towards the RAN, and one (usually public, or another private network) IP address towards the Core.

Currently, eUPF only supports configuring 1 global N3 local IP address through the "n3_address" config option.

As a result, when a PFCP client (an SMF, or osmo-hnbgw, or osmo-s1gw to mention some other examples) tries to setup a session to forward GTPU traffic from network A (RAN) to network B (Core), will result in eUPF forwarding the GTP-U packet to the Core but using the RAN configured IP as the GTPU packet local IP address.
Hence, either local routing will be wrong, or the Core will drop the received packet, or won't know how to route traffic back to it.

Describe the solution you'd like

3GPP TS 29.244 8.2.56 "Outer Header Creation" doesn't support setting the source IP address from PFCP (makes sense, this is UPF internal stuff).

Still, Create Far/Forwarding Parameters/Destination Interface = Core (3GPP TS 29.244 8.2.24) allows knowing the direction of the FAR.
Hence, we can set a local IP address in xdp update_gtp_tunnel() based on the destination interface.

Configuration wise, we could have a n3_ddress_core=A.B.C.D which would overwrite n3_address if set.
Alternatively, we can drop n3_address and have one for each type of interface in section 8.2.24.

Describe alternatives you've considered

Another option to explore would be to have a config option to let the kernel figure out the local IP address to use based on system routing table. I suspect it may be possible to get it when calling bpf_fib_lookup().

Additional context

Related code paths can be found below.

The local IP address of the GTPU packet to transfer is set in cmd/ebpf/xdp/n3n6_entrypoint.c handle_gtp_packet() / update_gtp_tunnel():

handle_gtp_packet() {
...
    // N9: Only outer header GTP/UDP/IPv4 is supported at the moment
    if (far->outer_header_creation & OHC_GTP_U_UDP_IPv4)
    {
        upf_printk("upf: [n3] session for teid:%u -> %u remote:%pI4", teid, far->teid, &far->remoteip);
        update_gtp_tunnel(ctx, far->localip, far->remoteip, 0, far->teid);
    }
...
}

static __always_inline void update_gtp_tunnel(struct packet_context *ctx, int srcip, int dstip, __u8 tos, int teid) {

    ctx->gtp->teid = bpf_htonl(teid);
    ctx->ip4->saddr = srcip;
    ctx->ip4->daddr = dstip;
    ctx->ip4->check = 0;
    ctx->ip4->check = ipv4_csum(ctx->ip4, sizeof(*ctx->ip4));
}

Currently, the "far->localip" from XDP is populated from "config.Conf.N3Address":

func main() {
...
	// Create PFCP connection
	pfcpConn, err := core.NewPfcpConnection(config.Conf.PfcpAddress, config.Conf.PfcpNodeId, config.Conf.N3Address, bpfObjects, resourceManager)
	if err != nil {
		log.Fatal().Msgf("Could not create PFCP connection: %s", err.Error())
	}

func NewPfcpConnection(addr string, nodeId string, n3Ip string, mapOperations ebpf.ForwardingPlaneController, resourceManager *service.ResourceManager) (*PfcpConnection, error) {
...
	n3Addr := net.ParseIP(n3Ip)
	if n3Addr == nil {
		return nil, fmt.Errorf("failed to parse N3 IP address ID: %s", n3Ip)
	}
	log.Info().Msgf("Starting PFCP connection: %v with Node ID: %v and N3 address: %v", udpAddr, nodeId, n3Addr)
...
	return &PfcpConnection{
...
		nodeAddrV4:        udpAddr.IP,
		n3Address:         n3Addr,
...
}

Call stack passing the value to the bpf:

HandlePfcpSessionEstablishmentRequest()
	composeFarInfo(far, conn.n3Address.To4(), ebpf.FarInfo{})
		farInfo.LocalIP = binary.LittleEndian.Uint32(localIp)
			mapOperations.NewFar(farInfo)
				internalId, err := bpfObjects.FarIdTracker.GetNext()
				bpfObjects.FarMap.Put(internalId, unsafe.Pointer(&farInfo))

So my proposal is to, instead of passing 1 address, pass an array of addresses (indexed by its pfcp interface). Otherwise, pass 2 address fields instead of 1, ie. one to use for "Access" and one for "Core".

[pirog-spb]

Hi @pespin,

Just to clarify. Did you mean I-UPF function here? So you requested different local IPs while forwarding GTP-U packets between N3 and N9?

-- BR, Alex

[pespin]

Hi @pirog-spb ,

sorry I'm still a bit new to all the 5G interfaces. We are actually missing those features to use eUPF co-located with a 3G osmo-hnbgw [1] and 4G osmo-s1gw [2] under the mentioned network setups, both supporting PFCP to manage a UPF to forward data plane also in GTP1U protocol.

I did a quick look through 3GPP TS 23.501 and indeed it seems there's already a naming interface distinction between N3 and N9 I didn't know about.
So yes, probably having a "n9_address" config field instead of the "n3_ddress_core" I was mentioning may make sense.
Then use that address when in "Outer Header Creation" + "Create Far/Forwarding Parameters/Destination Interface = Core" mode, aka when update_gtp_tunnel() is called inside xdp.

3GPP TS 29.244 (PFCP) also seems to have that distinction into account, though seems mostly related to reporting:

• Table 8.2.118-1: Interface Type value" "N9"
• 8.2.166 "GTP-U Path Interface Type"

[1] https://osmocom.org/projects/osmohnbgw
[2] https://gitea.osmocom.org/erlang/osmo-s1gw

[pespin]

I attach a sample pfcp session setup for the scenario explained above (we are using it now to do some benchmarking of eUPF forwarding GTPu traffic).
pfcp_session_gtpu_forwarding.pcapng.gz

I also have a draft patch to implement src ip address patching here: https://github.com/osmocom/eupf/commits/pespin/pr-n9/

[pespin]

By the way, I think no matter my patches are applied or not, forwarded IP/UDP/GTPU packets arriving at the next hop show up in wireshark with wrong UDP checksum and they are being dropped by the system (not reaching the userspace app socket).
So I think there may be some bug in the UDP checksum calculation in the xdp code.

[pespin]

So I think there may be some bug in the UDP checksum calculation in the xdp code.

Fixed here in my test env, using upf-benchmark [1] :
#580

With that fix and my patch in https://github.com/osmocom/eupf/commits/pespin/pr-n9/ Implementing n9_address, I can get GTPU traffic back and forth properly. Let me know what you think about my n9_address proposal and I can submit a PR.

[1] https://gitea.osmocom.org/cellular-infrastructure/upf-benchmark

[pespin]

MR merged, closing ticket.