fix: refine filetype detection

[scanny]

Summary
Fixes a bug where a CSV file with asserted content-type application/vnd.ms-excel was incorrectly identified as an XLS file and failed partitioning.

Additional Context
The content_type argument to partitioning is often authored by the client system (e.g. Unstructured SDK) and is both unreliable and outside the control of the user. In this case the .csv -> XLS mapping is correct for certain purposes (Excel is often used to load and edit CSV files) but not for partitioning, and the user has no readily available way to override the mapping.

XLS files as well as seven other common binary file types can be efficiently detected 100% of the time (at least 99.999%) using code we already have in the file detector.

• Promote this direct-inspection strategy to be tried first.
• When DOC, DOCX, EPUB, ODT, PPT, PPTX, XLS, or XLSX is detected, use that file-type.
• When one of those types is NOT detected, clear the asserted content_type when it matches any of those types. This prevents the problem seen in the bug where the asserted content type was used to determine the file-type.
• The remaining content_type, guess MIME-type, and filename-extension mapping strategies are tried, in that order, only when direct inspection fails. This is largely the same as it was before.
• Fix bug/check for magic library availability doesn't appear to be correct #3781 while we were in the neighborhood.
• Fix bug/file type detection fallback strategy not working #3596 as well, essentially an earlier report of bug/check for magic library availability doesn't appear to be correct #3781.

[cragwolfe]

worth adding a test for the specific bug, "CSV file with asserted content-type application/vnd.ms-excel was incorrectly identified as an XLS file"?

[scanny]

worth adding a test for the specific bug, "CSV file with asserted content-type application/vnd.ms-excel was incorrectly identified as an XLS file"?

@cragwolfe Oh yes, absolutely, that was the first one I added :) it's this one, named a little differently as test_it_ignores_asserted_XLS_content_type_when_file_is_CSV():
2d15fea#diff-d88874605ea6403a7c07fd5ebb3bf5d795888e7fb2ecd90f15e60ef2300901a5R364

Line 364 in ‎test_unstructured/file_utils/test_filetype.py if the link doesn't work quite right.

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ _ArrayMemoryError: Unable to allocate 23.9 GiB for an array with shape (80032, 80032) and data type float32 /general/v0/general View Issue

_{Did you find this useful? React with a 👍 or 👎}