catching exceptions

[christophsturm]

I am trying out kotlin power assert in the test suite of failgood, and I'm currently using it for all my new tests as an experiment.

its great for asserting, but unit tests also need "safe traversal":

https://github.com/failgood/failgood/blob/5c8a0f527083e8bccce5e6dcba7eb7b648a63361/failgood/src/test/kotlin/failgood/functional/ErrorHandlingTest.kt#L46
here I am using assertNotNull:

// execute the suite, then:
            val failedContext = assertNotNull(suiteResult.failedContexts.singleOrNull())
            assert(failedContext.context.name == "root")
            assert(
                failedContext.failure.message
                    ?.contains(
                        "Trying to create a test in the wrong context. Make sure functions that create tests have " +
                            "ContextDSL as receiver"
                    ) == true
            ) { failedContext.failure.stackTraceToString() }

but it would be much cooler if i could instead write the first line as

            val failedContext = suiteResult.failedContexts.single()

and power assert would wrap it with a try / catch and treat the exception just like a failed assertion, outputting the value of suiteResult.failedContexts.

maybe the whole assertion part of the unit test could be wrapped in a block, lets call it then (from given/when/then)

then {
            val failedContext = suiteResult.failedContexts.single()
            assert(failedContext.context.name == "root")
            assert(
                failedContext.failure.message
                    ?.contains(
                        "Trying to create a test in the wrong context. Make sure functions that create tests have " +
                            "ContextDSL as receiver"
                    ) == true
            ) { failedContext.failure.stackTraceToString() }
}

(or
each line in the then block is either an assertion or an assignment for navigation. and everything is treated as one big soft assert.

[bnorm]

I like the idea, and I'll definitely think about if there's a good way to do it, but exceptions present an interesting challenge for 2 reasons:

1. What to do with the actual diagram? Right now when the diagram message is built it is passed to an overload of the same function so it can be included directly in any thrown exception. If we're catching an exception, we would need to re-throw the exception and update the message or duplicate the exception in some way. That can get really messy, especially for generated code.
2. How to know what to diagram? Exceptions can occur at pretty much any time. We would need to keep track of everything we've seen so far in a data structure of some kind and then query it during the catch clause. Right now we just generate separate code paths with hardcoded messages. It's a completely new design for the library but something I've kind of thought about before.

One potential short term solution could be to go the then code block route and replace !! within the code block with a manual NPE and custom diagram error.

then {
  val failedContext = suiteResult.failedContexts.single()!!
  ...
}

But this also introduces a support library which I've been trying to avoid. And it kind of introduces something which might not have a lot of longterm value if something more generic is introduced.

Anyways, some ramblings on the idea.

[christophsturm]

can you maybe explain how the diagramming works? I mean how does KPA know all the intermediate values when an error occurs?
I thought that it has to rewrite the code anyway to keep track of the values and when it does that it could as well wrap every method call with a try catch.

another thought: maybe the block could be rewritten with with:

then {
            with(suiteResult.failedContexts.single()) {
                 assert(context.name == "root")
                 assert(
                     failure.message
                        ...
                  ) { failedContext.failure.stackTraceToString() }
           }
}

and with gets a special treatment like assert.
the main purpose of the then block is to limit all the rewriting magic to a scope.

[bnorm]

The digram engine works be creating an expression tree from the parameters to the function called. Each node of the tree is stored in a temporary variable and branches in the expression (boolean combinations) are transformed into flattened or nested if blocks. As you suggest we could wrap each temporary variable expression in a try-catch.

But this only solves the case when we're transforming the parameters of a function. If we want to do this transformation on an entire block of code, basically every little operation needs to be wrapped in a try-catch. This could be a serious amount of overhead. And we'd also want to avoid catching exceptions which already contain diagrams; like if assert is being transformed already.

Let's say we figure all of that out though. When an exception is thrown, then what? Where does the diagram string go? For the code to remain correct after the transformation the exact same exception must be thrown so supporting custom exceptions would be very difficult. Would adding a suppressed exception be enough?

If you have ideas how these things can be solved - I'm definitely open to them - I'm just not seeing a clear path forward yet with this idea.

[christophsturm]

maybe we should just try to find a small step we can take in that direction, and maybe that small step is catching exceptions inside the instrumented methods.

for example in this snippet:

import kotlin.test.assertNotNull

data class C1(val name:String)
data class Context(val context:C1)
data class SR(val failedContexts:List<Context>)

with(sr().failedContexts.single()) {
    assert(context.name == "root")
    assert(
        false
    ) { context.name }
}

fun sr() = SR(listOf(Context(C1("blah"))))

the with could be transformed to

with(assertNotNull(runCatching {sr()}.getOrNull()?.runCatching { failedContexts.single() }?.getOrNull())) 

or with 2 helper functions:

fun <T> safe(function: () -> T) = try {
    function()
} catch (e: Throwable) {
    null
}

fun <T, R> T.safe(function: T.() -> R) = try {
    function()
} catch (e: Throwable) {
    null
}
with(assertNotNull(safe { sr() }?.safe { failedContexts.single() })) {

[TWiStErRob]

@bnorm just for then {} you don't need a support library, just a configurable FQFN with a specific signature requirement.