Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode escape #354

Merged
merged 8 commits into from
Dec 27, 2020
Merged

Unicode escape #354

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f922d9a
[#334] parse and unparse tests
dariodsa Nov 14, 2020
d1b48b7
removed parsing and unparing tests
dariodsa Nov 18, 2020
0e81ac8
[#334] showUnicodeText
dariodsa Nov 18, 2020
7587a21
[#334] escaping unicode character as well as regular characters
dariodsa Dec 8, 2020
0349ff4
[#334] resolved issue with escaping regular unescaped chars
dariodsa Dec 9, 2020
2aace95
added tests, but they are not in use
dariodsa Dec 9, 2020
52d68c5
examples.hs revert to original content
dariodsa Dec 9, 2020
fb68968
[#334] changes requested by chshersh
dariodsa Dec 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions src/Toml/Type/Printer.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,18 @@ module Toml.Type.Printer
) where

import Data.Bifunctor (first)
import Data.Char (isAscii, ord)
import Data.Coerce (coerce)
import Data.Function (on)
import Data.HashMap.Strict (HashMap)
import Data.List (sortBy)
import Data.List (sortBy, foldl')
import Data.List.NonEmpty (NonEmpty)
import Data.Semigroup (stimes)
import Data.Text (Text)
import Data.Time (ZonedTime, defaultTimeLocale, formatTime)

import Text.Printf (printf)

import Toml.Type.AnyValue (AnyValue (..))
import Toml.Type.Key (Key (..), Piece (..))
import Toml.Type.PrefixTree (PrefixMap, PrefixTree (..))
Expand All @@ -40,6 +43,7 @@ import qualified Data.List.NonEmpty as NonEmpty
import qualified Data.Text as Text



{- | Configures the pretty printer.

@since 0.5.0
Expand Down Expand Up @@ -161,7 +165,7 @@ prettyKeyValue options i = mapOrdered (\kv -> [kvText kv]) options . HashMap.toL
valText (Bool b) = Text.toLower $ showText b
valText (Integer n) = showText n
valText (Double d) = showDouble d
valText (Text s) = showText s
valText (Text s) = showTextUnicode s
valText (Zoned z) = showZonedTime z
valText (Local l) = showText l
valText (Day d) = showText d
Expand All @@ -171,6 +175,23 @@ prettyKeyValue options i = mapOrdered (\kv -> [kvText kv]) options . HashMap.toL
showText :: Show a => a -> Text
showText = Text.pack . show


-- | Function encodes all non-ascii characters in TOML defined form using the isAscii function
showTextUnicode :: Text -> Text
dariodsa marked this conversation as resolved.
Show resolved Hide resolved
showTextUnicode text = Text.pack $ show finalText
where
xss = Text.unpack text
finalText = foldl' (\acc (ch, asciiCh) -> acc ++ getCh ch asciiCh) "" asciiArr

asciiArr = zip xss $ asciiStatus xss

getCh :: Char -> Bool -> String
getCh ch True = [ch] -- it is true ascii character
getCh ch False = printf "\\U%08x" (ord ch) :: String -- it is not true ascii character, it must be encoded

asciiStatus :: String -> [Bool]
asciiStatus = map isAscii

showDouble :: Double -> Text
showDouble d | isInfinite d && d < 0 = "-inf"
| isInfinite d = "inf"
Expand Down
9 changes: 9 additions & 0 deletions test/Test/Toml/Gen.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,13 @@ genUniHex8Color = do
hex <- genDiffHex 8
pure . Text.pack $ "\\U" ++ hex

-- | Generates some unescaped unicode string
genUnicodeChar :: Gen Text
genUnicodeChar = Gen.element
[ "č", "ć", "š", "đ", "ž", "Ö", "ё"
, "в", "ь", "ж", "ю", "ч", "ü", "я"
]

-- | Generates text from different symbols.
genText :: Gen Text
genText = genNotEscape $ fmap Text.concat $ Gen.list (Range.constant 0 256) $ Gen.choice
Expand All @@ -307,8 +314,10 @@ genText = genNotEscape $ fmap Text.concat $ Gen.list (Range.constant 0 256) $ Ge
, genPunctuation
, genUniHex4Color
, genUniHex8Color
--, genUnicodeChar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be uncommented back?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

genUnicodeChar is the function that I wrote myself thinking that I might be helpful to add it into the testing phase. But if I enable it, the test will fail.
The reason why it fails is mentioned above but I can repeat it. Character č is transformed into \U0000010d but decoding it back will give \U0000010d again, not č so the encode . decode tests will fail.
Currently, tests are only include escaped unicode characters in encode . decode phase.

]


genString :: Gen String
genString = Text.unpack <$> genText

Expand Down