Reconnect if IP is not accessible

[patrickelectric]

From discord
I’m not a web developer, so I haven’t really dug into the codebase to fully understand the implementation yet, but I definitely plan to in the future to be able to contribute more effectively.

Right now, I’m testing mavlink-server in one of our applications and noticed some behavior I wanted to clarify. I’m not sure if I’m missing something, so any insights would be appreciated!

Here’s my setup:

• CubeOrange connected to an Raspberry Pi via serial.
• The Pi is connected via Ethernet to two 4G routers (running VPN).
• I want mavlink-server to forward the MAVLink data to 4 separate UDP endpoints.

I’ve configured the Pi to start up mavlink-server on boot with this command:

mavlink-server serial:/dev/ttyS0:115200 udpclient:192.168.0.21:11999 udpclient:192.168.0.21:10001 udpclient:192.168.0.22:10002 udpclient:192.168.0.22:21999

The issue:

• When I power cycle the device, the Cube and Pi boot up quickly, but the 4G/VPN routers take a minute or two to get online.
• Once the routers are online (confirmed they’re up), mavlink-server doesn’t reconnect to my GCS everytime sometimes one endpoint connects and other doesn't.
• If I manually restart mavlink-server, it immediately connects all endpoints and works fine. I have attached 2 logs 00 was on power cycle where one connection was successful and other wasn't, and log 01 was after restart mavlin-server service which worked fine.

I understand I can write a script to wait for the 4G to come online before starting mavlink-server, but I’m curious if there’s any built-in behavior for retrying connections that I might be missing. Or is there a setting I should configure to handle this better?

Also, I’ve tested the exact same setup with MAVProxy using the following command:

mavproxy.py --master=/dev/ttyS0:115200 --out 192.168.0.21:11999 --out 192.168.0.21:10001 --out 192.168.0.22:10002 --out 192.168.0.22:21999

The same behavior persists with MAVProxy.

[loki077]

@patrickelectric did you had time to look into this issue?

[patrickelectric]

I just did a simple test and it appears to be working, I connected to another mavlink-server running on the vehicle, and disconnected my ethernet cable and reconnected back again, and it works.
@loki077 can you provide an example between two machines where the issues is replicable ? Also, try the master version if possible.

[joaoantoniocardoso]

maybe try setting the network interface down?

[loki077]

Hey @patrickelectric,

I’ve set up a Mavlink server on an RPi where the Cube is connected over serial. The RPi is connected to two routers via a switch.
RPi IP: 192.168.0.11

I just installed the Mavlink server from the master branch and used the binaries from CI. However, I’m running into CRC failures when I execute the following:

mavlink-server serial:/dev/ttyS0:115200 udpclient:192.168.0.6:14550

Here’s the output log for reference:

 mavlink-server serial:/dev/ttyS0:115200 udpclient:192.168.0.6:14550
2024-11-13T07:01:23.132009Z  INFO main ThreadId(01) src/lib/logger.rs:172: mavlink-server, version: 0.1.0-9aca3175d3c313d7eab417b2506691bdecc740ea, build date: 2024-12-20
2024-11-13T07:01:23.132404Z  INFO main ThreadId(01) src/lib/logger.rs:179: Starting at 2024-11-13T07:01:23
2024-11-13T07:01:23.132592Z DEBUG main ThreadId(01) src/lib/logger.rs:183: Command line call: mavlink-server serial:/dev/ttyS0:115200 udpclient:192.168.0.6:14550
2024-11-13T07:01:23.132840Z DEBUG main ThreadId(01) src/lib/logger.rs:184: Command line input struct call: Args {
    endpoints: [
        Serial {
            name: ArcSwapAny(
                "Serial",
            ),
            uuid: 74201462-a8c7-5349-aa0b-d3e641a0149b,
            port_name: "/dev/ttyS0",
            baud_rate: 115200,
            on_message_input: Callback,
            on_message_output: Callback,
            stats: RwLock {
                data: AccumulatedDriverStats {
                    name: "Serial",
                    driver_type: "Serial",
                    stats: AccumulatedDriverStatsInner {
                        input: None,
                        output: None,
                    },
                },
            },
        },
        UdpClient {
            remote_addr: "192.168.0.6:14550",
            name: ArcSwapAny(
                "UdpClient",
            ),
            uuid: 2a8a8950-e52e-5201-8587-ab14f30eef90,
            on_message_input: Callback,
            on_message_output: Callback,
            stats: RwLock {
                data: AccumulatedDriverStats {
                    name: "UdpClient",
                    driver_type: "UdpClient",
                    stats: AccumulatedDriverStatsInner {
                        input: None,
                        output: None,
                    },
                },
            },
        },
    ],
    verbose: false,
    web_server: 0.0.0.0:8080,
    enable_tracing_level_log_file: false,
    log_path: Some(
        "./logs",
    ),
    streamreq_disable: true,
    udp_server_timeout: 10,
    mavlink_system_id: 1,
    mavlink_component_id: 191,
    mavlink_heartbeat_frequency: 1.0,
    send_initial_heartbeats: false,
    mavlink_version: 2,
    default_api_version: 1,
}
2024-11-13T07:01:23.138368Z  INFO                 main ThreadId(01) src/lib/web/mod.rs:32: Running web server on address 0.0.0.0:8080
2024-11-13T07:01:23.143455Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 9760, calculated_crc: 40210 }
2024-11-13T07:01:23.153157Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 3693, calculated_crc: 29848 }
2024-11-13T07:01:23.156370Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 39807, calculated_crc: 28336 }
2024-11-13T07:01:23.242804Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 41820, calculated_crc: 60403 }
2024-11-13T07:01:23.274552Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 10073, calculated_crc: 64807 }
2024-11-13T07:01:23.274629Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 22909, calculated_crc: 22472 }
2024-11-13T07:01:23.279414Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 25087, calculated_crc: 43792 }
2024-11-13T07:01:23.282018Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 19585, calculated_crc: 57929 }

Interestingly, when I use MavProxy, everything works fine. Command:

mavproxy.py --master=/dev/ttyS0:115200 --out 192.168.0.6:14550

Output from MavProxy:

mavproxy.py --master=/dev/ttyS0:115200 --out 192.168.0.6:14550
Connect /dev/ttyS0 source_system=255
Log Directory:
Telemetry log: mav.tlog
Waiting for heartbeat from /dev/ttyS0
MAV> Detected vehicle 1:1 on link 0
online system 1
RTL> Mode RTL
AP: GCS Failsafe Off

Any ideas why the Mavlink server setup is failing while MavProxy works perfectly? Appreciate any pointers!

Rpi Os:

uname -m
armv7l

Installed from CI
'mavlink-server-armv7-unknown-linux-musleabihf'

[joaoantoniocardoso]

Hi @loki077 can you get us the following log?

RUST_LOG=debug,mavlink_codec=trace mavlink-server serial:/dev/ttyS0:115200 udpclient:192.168.0.6:14550 --enable-tracing-level-log-file

This should allow us to probe the protocol decoding, handled by mavlink-codec.

Also, could you upload the .tlog file from mavproxy? If mavproxy succeeded in decoding those messages, we can then use that as mavlink-server input and reproduce your problem.

Another way of reproducing the problem would be to use mavproxy to connect to serial, make mavlink-server connect to mavproxy via TCP or udp, and check if the errors happen.

Thanks