Scala-Native Support (1000USD Bounty) #156
Comments
|
I'll give this one a go 🙂 |
|
Just wanted to mention that I am still working on this and have not abandoned it! I have PRed |
|
A wee little update - the approach that sttp has taken unfortunately does not seem viable here; sttp simply does not allow any streaming with the curl backend which is how they got away with only using the libcurl |
|
@domaspoliakas sounds good! |
|
I have subscribed to this issue for a long time due to I also want to give a try in the beginning. But I'm a little worry about my junior skills in Scala Naive. Excited that @domaspoliakas is taking over the ticket. I still hope I could give a hand, or we could cooperate to solve some problems? Like above mentioned, we're currently trying to build Scala native API on top of Someone has tried to create a binding (https://github.com/fsat/scala-native-libcurl-bindings) years ago and obviously abandoned already. What's your opinion? Regards |
|
I personally do like the idea (and indeed you could add sttp to the list of libs that could benefit from such a thing here), but one downside is that it would add a dependency. This has a couple of drawbacks. The first is that requests-scala is a very minimalist library and practically does not add any transitive dependencies, which this would change a little. The second is that it would be a dependency outside of this organisation's control. Whether these are blockers or not is for @lihaoyi to decide. If you do get the green light - take a look at https://github.com/domaspoliakas/requests-scala/blob/scala-native-cross-build/requests/src-native/requests/curl.scala . I've already covered a chunk of the API, at least the things that I needed so far, which you could use as a starting point. I used sttp as a reference for that, so you'll notice that the style of the bindings is in many ways reminiscent of their implementaiton. Another thing to mention is that I of course treated this as an internal detail of requests-scala so it may need some cleanup in order to be published as a library of its own. |
|
I dont mind a third party dep as along as it is well maintained and stable. Most important thing is that the end to end workflow works; we can refactor the implementation however we like later |
|
You could potentially use curl bindings automatically generated by sn-bindgen |
|
Note that sn-bindgen produces Scala3-only bindings. Alternatively, lots of binding code can be copypasted from https://github.com/indoorvivants/sn-bindgen-examples/blob/main/example-curl/src/main/scala/generated/curl.scala#L3113 and adapted to fit Scala 2. It should be a lot easier than handcrafting the entire API |
|
The scala-3 requirement was the main reason for hand-crafting the bindings in this case, although I have used the generated bindings as a guide for some of the trickier parts of the API. The API is actually fairly small in the end, and I don't think I'll be adding anything to the bindings at this point. I'll take this opportunity to mention that parts of the test suite work already, so there is light at the end of the tunnel. There's still a bit of work before all of it is ok (the looming threat is still |
|
Sounds great @domaspoliakas ! |
|
@domaspoliakas any updates here? If not I'll be putting it up in the next set of bounties |
|
June was unfortunately a tough month to find time for this and I apologise for disappearing as I have. I have been working on this again as of last weekend however. Simple functionality is working at the moment, but there are still quite a few things left:
That last one has eaten quite a lot of my time so far with unfortunately not much to show for it. Generally uploads seem to work (some hand testing and some of the test suite corroborate this), but so far I am yet to find the cause or the right combination of flags to make the multipart test pass. I am happy to continue working on it now that I have time again, but if you would rather I passed the torch on to someone else - that is totally understandable. EDIT: got the multipart bug fixed now, so there's that |
|
Got it! If you're still involved then go ahead and keep working on it, just wanted to make sure it isn't dropped |
|
FYI for anyone reading, this bounty is still open in case anyone else wants to take a crack at this |
|
Sorry for disappearing once again. I'll officially step aside for this one. For those of you interested in picking this up - I think my WIP branch here could be a useful resource, with the most interesting bits being the curl bindings here and the actual implementation here. A lot of the basic tests (e.g. these) pass on this branch, but there's an issue somewhere that is causing this test to fail that I never managed to track down. Due to that and also some other missing pieces I think I would personally restart with a fresh implementation rather than try to fix that one, but I think some of the trickier bits (e.g. headerfunction and friends) work well and could be very useful for any future implementors. Lastly - I have spent many hours reading curl docs and its code to try and get this to work, and would be more than happy to answer any questions. Feel free to tag me here, or find me on discord (wisetree on discord) and fire away. There's a lot to curl, and I'd love to help springboard any new attempts at this. Finally - thank you for the opportunity to work on this one. It has proven to be too much for me, but I hope to see someone else succeed. |
|
No worries @domaspoliakas ! Thanks for taking a crack at it. Hopefully others will be able to build upon your work and move the ball forward |
|
I think leveraging the http4s-amber (src) for support of Scala Native is better than relying on curl in the long term. The
This route is more challenging, but once it's done, it also benefits the whole Scala Native community. What's more, both http client and server should work. What do you think? |
lihaoyi
changed the title
|
Bumping the bounty on this from 500 -> 1000USD |
|
IMO the best way to approach this would to implement the Java standard library |
|
I'm trying to implement this feature by following @domaspoliakas's attempts, which are introducing This is also a message to clarify my interesting of bounty, but I'll probably be slow, like 1 or 2 months. Anyone others cloud also give their own PR :). |
Hi, Haoyi, This PR will be part of my work to introduce native supports to `requests-scala` [#156]. I try to update current build dependencies to make sure that current tool chains are compatible with Scala Native and current implementations before introducing other breaking changes. Here are some notable changes: 1. Scala 2.11 must be deprecated because `geny` has deprecated Scala 2.11 since 1.1.0 2. Add Java 21 LTS to GitHub CI for testing job 3. All test cases passed in my local env. 4. But mima binary checks failed. I'm not very sure if it's fine or not. More details attached here Local testing results after changes ```console $ ./mill -i __.publishArtifacts + __.test [build.mill-57/61] compile [build.mill-57] [info] compiling 1 Scala source to /home/lqhuang/Git/requests-scala/out/mill-build/compile.dest/classes ... [build.mill-57] [info] done compiling [418/420] requests[2.13.15].test.test [418] [416/420] requests[3.6.2].test.test [416] [420/420] requests[2.12.20].test.test [420] [420/420] =============================================================== __.publishArtifacts + __.test ================================================================== 6s ``` report binary issues ```console $ ./mill -i __.mimaReportBinaryIssues [183/185] requests[2.13.15].mimaReportBinaryIssues [184/185] requests[2.12.20].mimaReportBinaryIssues ... [183] Found 52 issue when checking against com.lihaoyi:requests:0.7.0 [184] Found 52 issue when checking against com.lihaoyi:requests:0.7.0 ... [185/185] ================================================================= __.mimaReportBinaryIssues ==================================================================== 2s 3 tasks failed requests[2.12.20].mimaReportBinaryIssues Failed binary compatibility check! Found 206 potential problems requests[2.13.15].mimaReportBinaryIssues Failed binary compatibility check! Found 206 potential problems requests[3.6.2].resolvedMimaPreviousArtifacts scala.MatchError: Failure( Resolution failed for 1 modules: -------------------------------------------- com.lihaoyi:requests_3:0.6.7 not found: /home/lqhuang/.ivy2/local/com.lihaoyi/requests_3/0.6.7/ivys/ivy.xml not found: https://repo1.maven.org/maven2/com/lihaoyi/requests_3/0.6.7/requests_3-0.6.7.pom -------------------------------------------- ``` Any feedback and suggestion is appreciated! Regards, Lanqing ---- Updates: After refactoring `build.mill`, now binary check passed in my local env, too. ``` $ ./mill clean && ./mill -i __.mimaReportBinaryIssues [build.mill-64/68] compile [build.mill-64] [info] compiling 1 Scala source to /home/lqhuang/Git/requests-scala/out/mill-build/compile.dest/classes ... [build.mill-64] [info] done compiling [1/1] clean [1/1] ============================== clean ============================== [197/212] mill.scalalib.ZincWorkerModule.worker [197] Compiling compiler interface... [200/212] requests.jvm[3.3.4].compile [199/212] requests.jvm[2.13.15].compile [199] [info] compiling 7 Scala sources to /home/lqhuang/Git/requests-scala/out/requests/jvm/2.13.15/compile.dest/classes ... [200] [info] compiling 7 Scala sources to /home/lqhuang/Git/requests-scala/out/requests/jvm/3.3.4/compile.dest/classes ... [199] [warn] 4 deprecations (since 0.9.0) [199] [warn] 4 deprecations (since 2.13.0) [199] [warn] 8 deprecations in total; re-run with -deprecation for details [199] [warn] 3 feature warnings; re-run with -feature for details [199] [warn] four warnings found [199] [info] done compiling [200] [warn] there were 3 feature warnings; re-run with -feature for details [200] [warn] there were 6 deprecation warnings; re-run with -deprecation for details [200] [warn] two warnings found [200] [info] done compiling [204/212] requests.jvm[2.13.15].mimaReportBinaryIssues [208/212] requests.jvm[3.3.4].mimaReportBinaryIssues [204] Scanning binary compatibility in /home/lqhuang/Git/requests-scala/out/requests/jvm/2.13.15/compile.dest/classes ... [208] Scanning binary compatibility in /home/lqhuang/Git/requests-scala/out/requests/jvm/3.3.4/compile.dest/classes ... [204] Binary compatibility check passed [198/212] requests.jvm[2.12.20].compile [198] [info] compiling 7 Scala sources to /home/lqhuang/Git/requests-scala/out/requests/jvm/2.12.20/compile.dest/classes ... [208] Binary compatibility check passed [198] [warn] 6 deprecations (since 0.9.0); re-run with -deprecation for details [198] [warn] three feature warnings; re-run with -feature for details [198] [warn] two warnings found [198] [info] done compiling [212/212] requests.jvm[2.12.20].mimaReportBinaryIssues [212] Scanning binary compatibility in /home/lqhuang/Git/requests-scala/out/requests/jvm/2.12.20/compile.dest/classes ... [212] Binary compatibility check passed [212/212] ============================== __.mimaReportBinaryIssues ============================== 13s ``` And now if you access `https://self-signed.badssl.com/`, it literally returns a "404 Not Found". Moreover, it looks like the whole site `badssl.com` returns `404` (???? Shutdown or server error?) I update corresponding unit tests. ```console $ curl --insecure -vvv https://self-signed.badssl.com/ * Uses proxy env variable no_proxy == '.local,.internal,.arpa,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16' * Host self-signed.badssl.com:443 was resolved. * IPv6: (none) * IPv4: 104.154.89.105 * Trying 104.154.89.105:443... * Connected to self-signed.badssl.com (104.154.89.105) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / prime256v1 / rsaEncryption * ALPN: server accepted http/1.1 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=BadSSL; CN=*.badssl.com * start date: Dec 19 21:03:33 2024 GMT * expire date: Dec 19 21:03:33 2026 GMT * issuer: C=US; ST=California; L=San Francisco; O=BadSSL; CN=*.badssl.com * SSL certificate verify result: self-signed certificate (18), continuing anyway. * Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * using HTTP/1.x > GET / HTTP/1.1 > Host: self-signed.badssl.com > User-Agent: curl/8.5.0 > Accept: */* > < HTTP/1.1 404 Not Found < Server: nginx/1.10.3 (Ubuntu) < Date: Tue, 07 Jan 2025 10:41:04 GMT < Content-Type: text/html < Content-Length: 178 < Connection: keep-alive < <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.10.3 (Ubuntu)</center> </body> </html> * Connection #0 to host self-signed.badssl.com left intact ``` Current notable changes: 1. Scala 2.11 must be deprecated because `geny` has deprecated Scala 2.11 since 1.1.0 2. `Option.when(VcsVersion.vcsState().commitsSinceLastTag != 0)(VcsVersion.vcsState().lastTag).flatten` only appends `0.6.7`, so I replaced it with concrete release versions. 3. Add Java 21 LTS to GitHub CI for testing job 4. Add Scala 3.6.2 to GitHub CI to run testing job for Scala Next 5. test cases passed in my local env. 6. binary check passed in my local env --------- Co-authored-by: Li Haoyi <[email protected]> Co-authored-by: Ondra Pelech <[email protected]>
|
After reviewing current progress and status all above, I think @lihaoyi is right.
And it will benefit other Scala Native projects, too, by providing an optional replaceable plugin as Scala Native community wishes. And if there is better solution (whatever backend of I forked lolgab/scala-native-http-client-async into lqhuang/scala-native-http (Just clone and push again, but keep all history commits. It's a bit annoyed for me to see This is a stage report for my recent work and future schedules (without time commitments LOL) Thanks for prev attempts and hints from @domaspoliakas (it also may depend on the state of scala-native/scala-native#3921 while going further) and @lolgab Regards |
STTP Support Scala-Native via Curl. Requests-Scala should be able to support it via the same approach. We just need it to be wired into the Requests-Scala codebase, possibly refactoring out the actual making of the request into
src-jvm/andsrc-native/platform-specific code.Any platform-specific functionality that doesn't work on Scala Native should be called out, explained clearly, and their associated tests moved into platform-specific folders as well
To incentivize contribution, I'm putting a 1000USD bounty on resolving this ticket. This is payable via bank transfer, and at my discretion in case of ambiguity.
The text was updated successfully, but these errors were encountered: