Add Source Code Typing Tool #15211

Vici37 · 2024-11-21T05:10:54Z

This PR introduces a new crystal tool apply-types subcommand as described in this crystal forum post and initially implemented in this other repo as a stand alone tool. A tool that fully types a crystal file is obviously very tightly coupled with the language and compiler semantic, and so having this tool be part of the compiler itself feels like the correct direction.

That being said, I'm not 100% it's ready yet, but I do think it's ready enough at least to solicit feedback from anyone who is willing to provide it :)

Vici37 · 2024-11-21T05:12:08Z

spec/compiler/apply_types_spec.cr

+    OUTPUT
+  end
+
+  # it "parses, runs semantic, and types everything" do


Commenting out this spec for now - it runs prelude as well as try to type everything the test specific file (but that test file isn't part of this PR yet).

Vici37 · 2024-11-21T05:12:52Z

src/compiler/crystal/command/typer.cr

+# logic is in `crystal/tools/formatter.cr`.
+
+class Crystal::Command
+  private def typer


Entrypoint for the apply-types tool

Vici37 · 2024-11-21T05:17:10Z

src/compiler/crystal/tools/typer.cr

@@ -0,0 +1,526 @@
+module Crystal
+  class SourceTyper


In the source-typer repo, a lot of these classes / structs are broken into their own files. Precedence in the compiler seems to be to put a lot of these smaller bits into a single file if they can all be more or less self contained.

crysbot · 2024-11-21T05:36:29Z

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/exploring-the-compiler/7343/21

nobodywasishere · 2024-11-21T20:50:08Z

This is awesome!

Vici37 · 2025-01-11T17:21:43Z

src/compiler/crystal/command.cr

@@ -39,6 +39,7 @@ class Crystal::Command
    Usage: crystal tool [tool] [switches] [program file] [--] [arguments]

    Tool:
+        apply-types              add type restrictions to all untyped defs and def arguments


Name suggestions welcome! Another option I thought of is apply-restrictions as that feels more accurate, but also longer to type.

bake-types may also work, apply-restrictions is also pretty good

crysbot · 2025-01-23T17:00:19Z

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/rfc-variable-declaration-syntax-using/7623/35

Implement typer writing (inefficiently) Implement formatter, support multiple filename inputs Partial support for types, still need modules Update to latest source typer changes Add code comments Comment out broken spec for now Uncomment and fix final spec, add semantic / progress_tracker flags Remove focus: true (oops) Good ol' print debugging for windows CI failure Reimplement def visitor def locator matching for windows Back to print debugging Fix and support windows drive letters for root folders

…age when it happens

Vici37 · 2025-01-24T05:38:49Z

spec/compiler/apply_types_spec.cr

@@ -0,0 +1,580 @@
+require "./spec_helper"
+
+def run_source_typer_spec(input, expected_output,


The specs are written with a before (INPUT) -> after (OUTPUT) pattern, which should hopefully showcase all of the use cases covered by the tool. All of the specs but the last one skip the prelude run for performance.

Vici37 · 2025-01-24T05:40:07Z

src/compiler/crystal/tools/typer.cr

+      rets = {} of String => String
+
+      @warnings.each do |warning|
+        puts "WARNING: #{warning}"


Warnings are cases where type restrictions weren't added because it crossed a line with the language syntax somehow (i.e. empty splat calls)

Vici37 · 2025-01-24T05:42:30Z

src/compiler/crystal/tools/typer.cr

+      # And now infer types of everything
+      semantic_node = program.semantic nodes, cleanup: true
+
+      # We might run semantic later in an attempt to resolve defaults, don't display those stats or progress


Per this comment, a method can be defined like:

def my_method(arg = "str")

In this case, semantic may run on the "str" (or whatever expression) in an attempt to figure out what it is. It's occasional that methods like these always get called with an argument that isn't the same type as the default, and then the compiler complains if the restriction ends up like:

def my_method(arg : Int32 = "str")

Vici37 · 2025-01-24T05:45:24Z

src/compiler/crystal/tools/typer.cr

+      ret
+    end
+
+    private def def_overrides_parent_def(type) : Hash(String, String)


This method scans all defs in type and figures out the locations of all methods that override / are overridden between type and its ancestor's definitions. These methods should not have their types restricted in any additional way. This method is required as there's no syntatic way to indicate one method overrides another.

Vici37 · 2025-01-24T05:46:41Z

src/compiler/crystal/tools/typer.cr

+    end
+
+    # Generates a map of (parsed) Def#location => Signature for that Def
+    private def init_signatures(accepted_defs : Hash(String, Crystal::Def)) : Hash(String, Signature)


Workhorse method that generates the hash of location strings => signature for the def at that location

Vici37 · 2025-01-24T05:47:47Z

src/compiler/crystal/tools/typer.cr

+      end
+    end
+
+    def flatten_types(types : Array(Crystal::Type)) : Array(Crystal::Type)


Used to flatten unions into a single array, so dupe types can be deduplicated

Vici37 · 2025-01-24T05:49:17Z

src/compiler/crystal/tools/typer.cr

+    end
+
+    def type_name(type : Crystal::Type) : String
+      type.to_s.gsub(/:Module$/, ".class").gsub("+", "")


This was an interesting discovery: metatypes of classes can be created using the .class, but the syntax for creating a metatype of a module (also .class) isn't the same string representation that the compiler uses. The more you know

Vici37 · 2025-01-24T05:50:02Z

src/compiler/crystal/tools/typer.cr

+    # if there's a signature for a given def and those type restrictions are missing.
+    #
+    # All methods present are copy / paste from the original Crystal::Formatter for the given `visit` methods
+    class SourceTyperFormatter < Crystal::Formatter


Done typing, onto formatting!

Vici37 · 2025-01-24T05:51:06Z

src/compiler/crystal/tools/typer.cr

+    # A visitor for defs, oddly enough.
+    #
+    # Walk through the AST and capture all references to Defs that match a def_locator
+    class DefVisitor < Crystal::Visitor


DefVisitor, a crystal visitor for.... visiting defs. Used for capturing a list of all passed through defs that match a def_locator used when running the CLI command.

Vici37 commented Nov 21, 2024

View reviewed changes

straight-shoota marked this pull request as draft November 21, 2024 10:40

straight-shoota added topic:tools kind:feature labels Nov 21, 2024

Vici37 force-pushed the typer-tool branch from 25531e1 to fd162cf Compare November 25, 2024 16:33

Vici37 mentioned this pull request Dec 8, 2024

Some improvement? Vici37/cr-source-typer#1

Open

Vici37 force-pushed the typer-tool branch from b2def8a to cb15269 Compare December 14, 2024 05:28

nobodywasishere mentioned this pull request Jan 9, 2025

Add parameter and return type restrictions to all stdlib public methods #15334

Open

Vici37 force-pushed the typer-tool branch from cb15269 to 06b86e7 Compare January 11, 2025 16:57

Vici37 commented Jan 11, 2025

View reviewed changes

Vici37 added 9 commits January 23, 2025 21:53

Update comments, add exclude flag, fix bug for keword named params

d47e0ba

Fix bug for method external_names

6a0ecf8

Fix bug for module metatype arg types. Add TODO

56d11b1

Flatten out types and union types

74b8556

Fix bug for Metaclass Virtualtypes (wow)

d61e2ce

Support default types, don't add types to overridden methods

acc6b9f

Support types from default args that are constants

4deeee6

Don't add restrictions to parent overridden methods, add warning mess…

d8b713d

…age when it happens

Vici37 force-pushed the typer-tool branch from b194dbf to d8b713d Compare January 24, 2025 05:36

Vici37 commented Jan 24, 2025

View reviewed changes

Vici37 marked this pull request as ready for review January 24, 2025 05:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Source Code Typing Tool #15211

Add Source Code Typing Tool #15211

Vici37 commented Nov 21, 2024

Vici37 Nov 21, 2024

Vici37 Nov 21, 2024

Vici37 Nov 21, 2024

crysbot commented Nov 21, 2024

nobodywasishere commented Nov 21, 2024

Vici37 Jan 11, 2025

nobodywasishere Jan 24, 2025

crysbot commented Jan 23, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

Vici37 Jan 24, 2025

		@@ -0,0 +1,580 @@
		require "./spec_helper"

		def run_source_typer_spec(input, expected_output,

Add Source Code Typing Tool #15211

Are you sure you want to change the base?

Add Source Code Typing Tool #15211

Conversation

Vici37 commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crysbot commented Nov 21, 2024

nobodywasishere commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crysbot commented Jan 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment