Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relaxed mode for parsing invalid MIBs #37

Open
Jellyfrog opened this issue Jun 3, 2022 · 3 comments
Open

Relaxed mode for parsing invalid MIBs #37

Jellyfrog opened this issue Jun 3, 2022 · 3 comments

Comments

@Jellyfrog
Copy link

Jellyfrog commented Jun 3, 2022

Librenms got a decent collection of MIBs from various vendors, some more broken than others.

It might be of interest to add a "relaxed mode" for common problems, such as the ones with invalid dates: unexpected "201506080000A" (expected <extutctime> ...)

$ git clone https://github.com/librenms/librenms.git /tmp/librenms
$ find /tmp/librenms/mibs/ -type f | xargs -P0 -n1 ./parse 1>/dev/null
2022/06/03 16:22:32 Parse file: /tmp/librenms/mibs/SNMPv2-TC-v1:121:1: unexpected token "SNMPv2-CONF-v1"
2022/06/03 16:22:33 Parse file: /tmp/librenms/mibs/allied/AT-DOS-MIB:72:11: unexpected "/" (expected "(" ...)
2022/06/03 16:22:35 Parse file: /tmp/librenms/mibs/eltek/VALERE-DC-POWER-MIB:2460:30: unexpected "bdtAlarmPresent" (expected "}")
2022/06/03 16:22:37 Parse file: /tmp/librenms/mibs/cisco/CISCOSB-PBR-MIB:13:22: unexpected "201506080000A" (expected <extutctime> ...)
2022/06/03 16:22:39 Parse file: /tmp/librenms/mibs/cisco/CISCOSB-ROUTEMAP-MIB:12:22: unexpected "201506080000A" (expected <extutctime> ...)
2022/06/03 16:22:39 Parse file: /tmp/librenms/mibs/imco/IMCO-BIG-MIB:12202:2: unexpected "aSupply1UnitAny" (expected "}")
2022/06/03 16:22:40 Parse file: /tmp/librenms/mibs/cisco/CISCOSB-SYSMNG-MIB:16:22: unexpected "201010310000a" (expected <extutctime> ...)
2022/06/03 16:22:43 Parse file: /tmp/librenms/mibs/endrun/TEMPUSLXUNISON-MIB:285:7: unexpected "1" (expected "}")
2022/06/03 16:22:43 Parse file: /tmp/librenms/mibs/aerohive/AH_TRAP_MIB:37:21: unexpected "8021" (expected "}")
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/riedo/NETTRACK-E3METER-CTR-SNMP-MIB:466:3: unexpected "1" (expected "}")
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/terra/TERRA-PRODUCTS-MIB:18:9: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/ekinops/EKINOPS-MGNT2-NMS-MIB:357:4: unexpected "undefinedError" (expected "}")
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/terra/TERRA-DEFINITIONS-MIB:20:9: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/transition/TN-OTDR-MIB:690:13: unexpected "1" (expected <ident>)
2022/06/03 16:22:45 Parse file: /tmp/librenms/mibs/transition/TN-LLDP-EXT-MIB:23:21: unexpected "2013030251930Z" (expected <extutctime> ...)
2022/06/03 16:22:47 Parse file: /tmp/librenms/mibs/juniper/JUNIPER-WX-MIB:241:1: unexpected token "JUNIPER-WX-GLOBAL-TC"
2022/06/03 16:22:49 Parse file: /tmp/librenms/mibs/riello/RIELLOMDU-MIB:216:109: unexpected "Imminent" (expected <assign> ...)
2022/06/03 16:22:49 Parse file: /tmp/librenms/mibs/poweralert/TRIPPLITE-12X:45:5: unexpected "Corrected spelling errors" (expected "DESCRIPTION" ...)
2022/06/03 16:22:51 Parse file: /tmp/librenms/mibs/eds/EDS-MIB:1352:23: unexpected "," (expected (<octetstring> | <objectidentifier> | <ident>) ...)
2022/06/03 16:22:57 Parse file: /tmp/librenms/mibs/hp/HP-SN-MIBS:74:1: unexpected token "HP-SN-AGENT-MIB"
2022/06/03 16:23:07 Parse file: /tmp/librenms/mibs/linksys/LINKSYS-SYSMNG-MIB:16:22: unexpected "201010310000a" (expected <extutctime> ...)
2022/06/03 16:23:08 Parse file: /tmp/librenms/mibs/linksys/LINKSYS-MODEL-MIB:3554:1: no match found for
2022/06/03 16:23:10 Parse file: /tmp/librenms/mibs/fs/GBNPlatformOAM-MIB:954:16: unexpected "10" (expected "}")
2022/06/03 16:23:16 Parse file: /tmp/librenms/mibs/dragonwave/HORIZON-ODU-MIB:204:7: unexpected "hc40-57-qpsk" (expected "}")
2022/06/03 16:23:18 Parse file: /tmp/librenms/mibs/delta/DeltaUPS-MIB:205:12: unexpected "3" (expected "}")
2022/06/03 16:23:18 Parse file: /tmp/librenms/mibs/fs/FS-MIB:18630:78: unexpected "cpemac" (expected "}")
2022/06/03 16:23:21 Parse file: /tmp/librenms/mibs/zyxel/ZYXEL-NWA-SERIES_v1-4-2:104:4: unexpected "success" (expected "}")
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/tplink/TPLINK-SYSINFO-MIB:115:16: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/proware/proware-SNMP-MIB:94:17: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/zyxel/ZYXEL-GS4012F-MIB:7529:22: unexpected "mandatory" (expected ("current" | "deprecated" | "obsolete") ...)
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/edgecos/ECS4100-52T-MIB:10879:4: unexpected "radiusServerType" (expected "}")
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/ibm/IBM-NetFinity-Text-Alert-MIB:132:1: no match found for
2022/06/03 16:23:25 Parse file: /tmp/librenms/mibs/ibm/IBMIROCAUTH-MIB:426:19: no match found for .
2022/06/03 16:23:28 Parse file: /tmp/librenms/mibs/zyxel/IES5206-MIB:10374:17: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:33 Parse file: /tmp/librenms/mibs/arris/d5/ARRIS-D5-SLOT-EXT-MIB:50:34: unexpected "(" (expected "END")
2022/06/03 16:23:37 Parse file: /tmp/librenms/mibs/perle/PERLE-IOLAN-SDS-MIB:657:6: unexpected "Description" (expected <assign> ...)
2022/06/03 16:23:38 Parse file: /tmp/librenms/mibs/junos/JUNIPER-WX-MIB:241:1: unexpected token "JUNIPER-WX-GLOBAL-TC"
2022/06/03 16:23:41 Parse file: /tmp/librenms/mibs/dlink/DLINKSW-NETWORK-ACCESS-MIB:487:44: unexpected "xffffffff" (expected ")")
2022/06/03 16:23:42 Parse file: /tmp/librenms/mibs/teldat/TELDAT-MON-INTERF-WLAN-MIB:42:10: unexpected "mandatory" (expected ("current" | "deprecated" | "obsolete") ...)
2022/06/03 16:23:43 Parse file: /tmp/librenms/mibs/dell/DELLEMC-OS10-CHASSIS-MIB:137:13: unexpected "os10ChassisProductBase" (expected "}")
2022/06/03 16:23:44 Parse file: /tmp/librenms/mibs/bdcom/NMS-CARD-SYS-MIB:70:23: unexpected "`" (expected (<octetstring> | <objectidentifier> | <ident>) ...)
2022/06/03 16:23:44 Parse file: /tmp/librenms/mibs/bdcom/NMS-CHASSIS:822:25: unexpected "TLB-modification-exception" (expected "}")
2022/06/03 16:23:45 Parse file: /tmp/librenms/mibs/cambium/800/CAMBIUM-PTP800-MIB:34:70: no match found for .
2022/06/03 16:23:45 Parse file: /tmp/librenms/mibs/calix/E7-TC:348:9: unexpected "ontAutoQuarantine" (expected "}")
2022/06/03 16:23:46 Parse file: /tmp/librenms/mibs/calix/E7-Calix-MIB:28:5: unexpected "E7OperStatus" (expected "FROM" ...)
2022/06/03 16:23:47 Parse file: /tmp/librenms/mibs/fortinet/FORTINET-FORTIGATE-MIB:9462:33: unexpected "1" (expected "}")
2022/06/03 16:23:48 Parse file: /tmp/librenms/mibs/brocade/FOUNDRY-SN-ROUTER-TRAP-MIB:439:1: no match found for
2022/06/03 16:23:48 Parse file: /tmp/librenms/mibs/calix/E5-120-MIB:7822:18: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:49 Parse file: /tmp/librenms/mibs/calix/E5-121-MIB:7932:18: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:50 Parse file: /tmp/librenms/mibs/screenos/NETSCREEN-TRAP-MIB:189:9: unexpected "low-memory" (expected "}")
2022/06/03 16:23:51 Parse file: /tmp/librenms/mibs/junose/broken/Juniper-Accounting-CONF:250:13: unexpected "DESCRIPTION" (expected <assign> ...)
2022/06/03 16:23:51 Parse file: /tmp/librenms/mibs/junose/broken/Juniper-ATM-1483-Profile-CONF:101:13: unexpected "juniAtm1483ProfileGroup4" (expected "STATUS" ...)
2022/06/03 16:23:51 Parse file: /tmp/librenms/mibs/junose/broken/Juniper-Internet-CONF:1593:18: unexpected "INCLUDES" (expected "{" ...)
2022/06/03 16:23:52 Parse file: /tmp/librenms/mibs/junose/JUNIPER-WX-MIB:241:1: unexpected token "JUNIPER-WX-GLOBAL-TC"
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-SYSTEM-ACCESS-MIB:528:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-SYSLOG-MIB:339:14: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-MPLS-RSVP-MIB:308:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-CPU-MAC-FILTER:195:14: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-SERVICE-POLICY-GROUP-MIB:354:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-CFM-MIB:2551:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-OSPF-MIB:332:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-QOS-MIB:166:7: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-SOFTWARE-MANAGEMENT-GROUP-MIB:580:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-MPLS-MIB:667:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-INTERFACE-MIB:1092:4: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ARP-MIB:248:4: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PON-MAC-GROUP-MIB:3155:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-ONTMANAGER-GROUP-MIB:178:5: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PM-MIB:487:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-MIB:1286:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ERPS-MIB:795:21: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-MPLS-PW-MIB:655:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-AUTO-RESET-MIB:505:14: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ENTITY-ALARM-MIB:287:22: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-REDUNDANCY-MIB:697:22: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PON-PROFILE-GROUP-MIB:3812:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-SLD-MIB:314:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PM-GROUP-MIB:3038:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-DOT1BRIDGE-MIB:200:24: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-NTP-MIB:196:21: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ACCESS-LIST-MIB:259:4: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/extreme/Brocade-TC:89:1: unexpected token "BROCADE-PRODUCTS-MIB"
2022/06/03 16:23:55 Parse file: /tmp/librenms/mibs/dasan/SLE-ISIS-MIB:29:4: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:23:56 Parse file: /tmp/librenms/mibs/dasan/DASAN-SWITCH-MIB:3511:5: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:56 Parse file: /tmp/librenms/mibs/dasan/DASAN-NOTIFICATION:1216:13: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:56 Parse file: /tmp/librenms/mibs/extreme/EXTREME-BASE-MIB:356:1: unexpected token "EXTREME-SYSTEM-MIB"
2022/06/03 16:23:57 Parse file: /tmp/librenms/mibs/dasan/SLE-VRRP-MIB:29:4: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:23:59 Parse file: /tmp/librenms/mibs/dasan/SLE-EPON-MIB:31:17: unexpected "" (expected <extutctime> ...)
2022/06/03 16:24:00 Parse file: /tmp/librenms/mibs/huawei/HUAWEI-SLOG-EUDM-MIB:113:19: unexpected "0" (expected "}")
2022/06/03 16:24:00 Parse file: /tmp/librenms/mibs/huawei/HUAWEI-MGMD-STD-MIB:747:5: unexpected "?" (expected "}")
2022/06/03 16:24:05 Parse file: /tmp/librenms/mibs/himoinsa/DISMUNTELv00-MIB:1294:1: unexpected token "cj"
2022/06/03 16:24:08 Parse file: /tmp/librenms/mibs/4rf/4RF-COMMON-TC:22:5: unexpected "Integer32" (expected "FROM" ...)
2022/06/03 16:24:08 Parse file: /tmp/librenms/mibs/4rf/4RF-APRISAXE-TC:111:45: unexpected "'E0'H" (expected <int>)

@sleepinggenius2
Copy link
Owner

sleepinggenius2 commented Jun 3, 2022

This isn't really practical with the current parser. I also don't know how many people might be using the library for linting, so I would want to be very careful about introducing any such workarounds without also including flags to be able to control them.

I have cloned the LibreNMS MIBs myself and have used them for some testing. CISCOSB-SYSMNG-MIB, for example, looks like it was correct until a bulk update ~10 months ago that messed up the revision dates. Unfortunately, it doesn't look like there have been checks added to their CI process to do linting or any other validation on the included MIB files, so they do get committed with errors, sometimes even regressions it seems.

I'm always happy to address bugs in the library for spec-compliant MIB files, but I would suggest in this scenario to open an issue against the LibreNMS repo to work on getting things corrected there.

@Jellyfrog
Copy link
Author

This isn't really practical with the current parser. I also don't know how many people might be using the library for linting, so I would want to be very careful about introducing any such workarounds without also including flags to be able to control them.

I'll rename the issue to reflect a bit better the goal of the issue.

I have cloned the LibreNMS MIBs myself and have used them for some testing. CISCOSB-SYSMNG-MIB, for example, looks like it was correct until a bulk update ~10 months ago that messed up the revision dates. Unfortunately, it doesn't look like there have been checks added to their CI process to do linting or any other validation on the included MIB files, so they do get committed with errors, sometimes even regressions it seems.

All (guessing 99%) comes directly, unmodified, from the vendor, this is more a reflection of the [sad] state of mibs out in the wild.
If we take the mentioned MIB, latest download I can find (https://software.cisco.com/download/home/284077831/type/283415684/release/1.4.10.06) contains;

$ grep LAST-UPDATED CISCOSBsysmng.mib
        LAST-UPDATED "201010310000a"

I'm always happy to address bugs in the library for spec-compliant MIB files, but I would suggest in this scenario to open an issue against the LibreNMS repo to work on getting things corrected there.

I'm on of the maintainers there, and I don't think we (LibreNMS) can/should modify vendor mibs

@Jellyfrog Jellyfrog changed the title Add workaround for common problems Relaxed mode for parsing invalid MIBs Jun 7, 2022
@sleepinggenius2
Copy link
Owner

I'll rename the issue to reflect a bit better the goal of the issue.

The new title I agree with. I've started to work on a major rewrite of the library's core that should start to help with some common errors in MIBs (like textual convention inheritance), but unfortunately I can't address parsing-related issues until I can replace the existing parser. As stated previously, I want to make sure that there is still a way to use the library as a linter, so I don't want to introduce any changes without still providing an option for spec compliance. The immediate goal it to reach parity with the original libsmi C library in what it is able to load (despite some errors), but my ultimate goal is to be able to load any MIB that Net-SNMP can. Unfortunately, my time is limited, so I can't commit to a timeline on this. A lot of work has already been completed, but not much on the parser yet.

All (guessing 99%) comes directly, unmodified, from the vendor, this is more a reflection of the [sad] state of mibs out in the wild. If we take the mentioned MIB, latest download I can find (https://software.cisco.com/download/home/284077831/type/283415684/release/1.4.10.06) contains;

$ grep LAST-UPDATED CISCOSBsysmng.mib
        LAST-UPDATED "201010310000a"

Yeah, the ones out in the wild are often pretty bad. Some vendors are certainly worse than others. I'm mostly happy when I can even get MIBs for a product, but too often even the latest available seem to be outdated when actually trying to walk a device.

I'm one of the maintainers there, and I don't think we (LibreNMS) can/should modify vendor mibs

I don't necessarily agree with that statement. If there's a change that can be made to correct a syntax issue and it allows for the MIB file to be more fully parsed or otherwise create more useful information, then I think it would be in everyone's best interest to do so. In my career, I've definitely done that for some vendor MIBs, even going so far as correcting types, enum values, adding missing objects, and so on, so that I can actually consume the MIB and do something useful with it. You definitely have to stay vigilant though, because I have had vendors fix some of those kind of bugs in newer firmware releases. That's why it's always important to try to use not just the correct MIBs, but the correct revisions for a particular device and firmware version. MIBs are supposed to be backwards compatible, but that's another thing that vendors all too often mess up as well.

Depending on your relationship with the vendor, you can try opening a ticket with them to "properly" correct the issue, but I have not had great success with most vendors going that route (even ones with active support contracts). Especially for older/unsupported products or companies that no longer exist, there's really little recourse to get the problem fixed at the source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants