Reference is not added to transformed border and shadow token

[lukasoppermann]

Hey,

I am not sure if I am missing something.

I use this in my settings for the file:

{
  // ...
 outputReferences: (token, platformOptions) =>
            outputReferencesFilter(token, platformOptions) && outputReferencesTransformed(token, platformOptions),
}

My border token looks like this:

  border: {
    default: {
      $value: {
        color: '{borderColor.default}',
        style: 'solid',
        width: '{borderWidth.default}',
      },
      $type: 'border',
    },
  }

The borderColor token is:

  borderColor: {
    default: {
      $value: '{base.color.neutral.6}',
      $type: 'color',
    },
  }

And the borderWidth token is:

  borderWidth: {
    default: {
      $value: '{borderWidth.thin}',
      $type: 'dimension',
    },
    thin: {
      $value: '1px',
      $type: 'dimension',
    },
  }

My transformer borderToCss looks like this:

export const borderToCss: Transform = {
  name: 'border/css',
  type: 'value',
  transitive: true,
  filter: isBorder,
  transform: (token: TransformedToken) => {
    const value = token.$value ?? token.value
    // if value === string it was probably already transformed
    if (typeof value === 'string') {
      return value
    }
    /* width | style | color */
    return `${value.width} ${value.style} ${value.color}`
  },
}

The expect output is:

--border-default: var(--borderWidth-default) solid var(--borderColor-default);

However, the actual output is:

--border-default: 0.0625rem solid #3d444d;

What am I doing wrong?

// Addition

I just noticed that it is working for shadows without a color and for text shorthands. I am beginning to expect it to be related to the color value.

--boxShadow-thick: inset 0 0 0 var(--borderWidth-thick);
  --text-title-shorthand-small: var(--text-title-weight-small) var(--text-title-size-small) / var(--text-title-lineHeight-small) var(--fontStack-sansSerif); 

[lukasoppermann]

If I look at this https://github.com/amzn/style-dictionary/blob/main/lib/utils/references/outputReferencesTransformed.js it seems like objects in tokens are not supported?

Is the token that gets passed to outputReferencesTransformed one of the referenced tokens (e.g. borderColor.default), or is it the entire token, e.g. my border.default token?

[jorenbroekema]

Yeah unfortunately this is a limitation of outputReferences.

When you are converting from Object to String, you no longer "know" which property of the object is where exactly in the string, so it's always fundamentally unsafe to then try to put back refs.

Let's take an example e.g. shadow where we can see this problem:

{
  offsetX: '{spacing.1}', // this resolves to 1px
  offsetY: '{spacing.11}', // this resolves to 11px
  blur: '21px',
  color: '#000'
}

This would be transformed to 1px 11px 21px #000, let's imagine we want to put back references now so that we can get from:
1px 11px 21px #000 to
var(--spacing-1) var(--spacing-11) 21px #000

You might see the problem here, we would take the original value (the object one), check for references, we will find two references, the first one is {spacing.1} which resolves to 1px, but this has multiple occurances in the final value... how do we know which one belongs to offsetX and which one belongs to offsetY? We didn't track this information when we stringified the object, Style Dictionary is not that smart so this information gets lost when a complex datatype gets transformed to a simpler/flatter one like Object -> string.

So, if we ignore this problem, we would take the resolved final value and replace occurences of 1px with the reference as a CSS var, which would result in:
var(--spacing-1) 1var(--spacing-1) 2var(--spacing-1) #000

Oops... that's not what we wanted...

So long story short, OutputReferencesTransformed prevents this from happening by not allowing you to output references for tokens that were originally objects but were simplified to strings, because it is unsafe unless you know precisely e.g. through the use of an Abstract Syntax Tree (AST) the exact locations where a object property is sitting inside the string value.
Theoretically we could make this library smart enough to do that but I don't think this is worthwhile. It's a lot of complexity for very little value. Imo it is better to only outputReferences for tokens that are simple / dumb references to other tokens (e.g. a component token that is just a reference to a semantic token), not for tokens that undergo complicated transformations.

[jorenbroekema]

I should note that we could also think outside the box here and allow:

• transforms to transform values even if they have references
• do not resolve references for certain tokens at any given time

which would theoretically let you end up with this value:
{spacing-1} {spacing-11} 21px #000

Which you could then format in the CSS formatter to:
var(--spacing-1) var(--spacing-11) 21px #000

So it would be theoretically possible if you could hook into the resolve references and deny certain tokens from resolving altogether, and relying on the format hook to convert the token refs to CSS vars.

The downside of that approach is that transitive transforms and regular transforms would never work on these tokens, since they normally have to wait for references to be resolved in order to do their work. Two use cases are Studio's transitive color modifers and resolve math transforms, but those can also be handled in CSS itself using color-mix() and calc(), but these have their own limitations as well. So even if this is made possible e.g. in v5 of Style Dictionary, you'll still find yourself struggling.

Hence why my personal recommendation is to not use outputReferences for anything other than tokens that are solely references (I call them dumb tokens). https://github.com/tokens-studio/lion-example?tab=readme-ov-file#tokens-structure documents this a bit for my example repo, where I do use outputReferences, but only for my component token layer which are pure references to the semantic layer (which is theme-dependent/dynamic, hence why CSS vars are useful).

[lukasoppermann]

Ahh, okay that does make sense. Is this why it is working if there is only one reference in the output?

The above issue could be solved by simply searching for 1px (including spaces). Probably more realistically adding a regex to also look for a training comma or semicolon.

Another option would be, that a transformer could provide some kind of map like this: {offsetX} {offsetY} {blur} {color} and than it can be resolved later with references.

I understand that this could be an issue, but is there a way that I can implement it for myself?
I could not just create a custom version of the outputReferencesTransformed because this does not output the token, only true or false, right? So where is the actual exchange happening?

Or would I do what you seem to suggest above, that I just replace the resolved value with the reference in my border transformer?

[jorenbroekema]

What about:

{
  offsetX: '{spacing.foo}', // this resolves to 1px
  offsetY: '{spacing.bar}', // this also resolves to 1px
  blur: '1px',
  color: '#000'
}

Even if we had a map {offsetX} {offsetY} {blur} {color}, there are still scenarios that we can theory-craft where this system would fail e.g. when there aren't convenient space separators in the stringified value.

It feels like every time I dive deep into this topic and try to come up with solutions, I find more edge cases that aren't covered and it feels like putting bandages on open wounds.

Whereas if you rethink the way you use references through your token taxonomy/structure and selectively output refs only for dumb tokens, all of these issues can easily be prevented.

That said, outputReferences is very flexible now as you can pass in your own function and selectively enable/disable it, but you cannot dictate unfortunately how the refs are outputted, only whether they are or not for a given token.

For you to completely control also the "how" would mean you'd have to create a custom format and put in place your own outputReferences feature. For inspiration, the createPropertyFormatter is the most sophisticated example of outputReferences that's built-in for Style Dictionary, so for example for the css/variables format:
https://github.com/amzn/style-dictionary/blob/main/lib/common/formats.js#L195 which uses
https://github.com/amzn/style-dictionary/blob/main/lib/common/formatHelpers/formattedVariables.js#L59 which is a small wrapper around
https://github.com/amzn/style-dictionary/blob/main/lib/common/formatHelpers/createPropertyFormatter.js#L176-L247 which is where the magic happens

Edit: no, I think if you replace the value with the references in your border transformer, you will still end up with the same problem because in order to put back refs in the createPropertyFormatter helper, it refers to the token's original value which is still the object. What I meant is that you could have another layer of abstraction in your tokens itself:

• token "border" refers to "{border2}" -> this will output a CSS var safely
• border2 uses the object notation and refers to {borderwidth} and border {bordercolor}. this will get stringified, so for this token you will not get a CSS var

[lukasoppermann]

That said, outputReferences is very flexible now as you can pass in your own function and selectively enable/disable it, but you cannot dictate unfortunately how the refs are outputted, only whether they are or not for a given token.

Does this mean I could try to enable it and see if the output works? Or will this definitely not give me the desired result?

Whereas if you rethink the way you use references through your token taxonomy/structure and selectively output refs only for dumb tokens, all of these issues can easily be prevented.

Maybe I am not getting what you mean, but this sounds like not using tokens to the max because of a technical issue? I want references so that I can adjust them on the fly (in the browser).

I understand that offering this solution as the tool is problematic, because you have to deal with all the edge cases. But as a consumer, I may be able to avoid all the edge cases, so it would be fine.

[jorenbroekema]

Does this mean I could try to enable it and see if the output works? Or will this definitely not give me the desired result?

In the particular use case you mentioned, I think it would work, but the odds that you will run into cases where it creates invalid CSS is pretty high, it depends on a few factors but comes down to how many transitive transforms are applied to your tokens that can cause the exact issue I outlined where the find & replace may make mistakes

not using tokens to the max because of a technical issue

It is a fundamental issue: One entity cannot be a reference to another entity and yet simultaneously be a transformed or otherwise processed or computed variant of itself. Once you do the transformation, the reference is lost because you have just diverged from the referenced value.

We have to think outside the box quite a bit in order to work around that problem. For example, if you would output refs first, and then do the transformation from object to string, then you're fine.

So perhaps the lifecycle should be:

1) pre-transforms (value-changing, can be transitive) / resolve refs
2) output refs (if possible) 
3) post-transforms (syntax/formatting only!)
4) format (handles {ref} -> var(--ref) )

That way, if a pre-transform changes the value, outputting refs will still be disallowed due to the fundamental issue, but your border object to string transformation can just be done at the very end, after all, it's just a syntax-only transformation, it doesn't change the inherent value of the token, it just reorders it or stringifies it in your case. Similar to converting from hex to rgb, it doesn't change the value, just the syntax.

It's something I would be open to playing with, but completely rethinking this part of the lifecycle is very costly even for v5 milestone. I just don't know if I'm convinced we really need to do this when it is my opinion that you should never end up in a spot where you should need this.

I want references so that I can adjust them on the fly (in the browser)

Yeah, my point is that you should adjust only 1 layer of your tokens on the fly (e.g. on theme switch), usually your semantics, and have everything else reference those. "everything else" is what you want to output references for, but the semantic layer of tokens that are "smart" (e.g. can contain math, color modifiers, object tokens with properties that contain/are references) doesn't benefit from outputting as CSS variables unless the thing that it references is also dynamic.

In summary, usually people build their tokens in this way:

• a layer of primitives, core, global tokens, think of spacer scale, color palettes
• a layer of semantic tokens that refer to the primitives, this layer is often dynamic through theming, meaning that your background accent color primary may be referring to colors.red.500 in one theme, but colors.blue.500 in another, and there can be multiple sublayers here for example, it may be colors.blue.800 in dark mode but colors.blue.300 in light mode.
• a layer of component or option/semantic tokens that are mostly just pure references to the semantic dynamic layer, meaning that you benefit from outputting references here, so you can keep the CSS exactly the same, and through CSS variable references the rendered values change when the active theme changes.

There's many variations of this but it always tends to come down to a form static -> dynamic -> static and only the final static layer really benefits from using CSS variables, at least that has been my observation looking at a lot of people's tokens over the past year.

[jorenbroekema]

Btw I'm open to having my view changed on all this but I'd have to see more clearly why outputting refs is so important for these tokens that imo wouldn't really benefit from it

[lukasoppermann]

In summary, usually people build their tokens in this way:

• a layer of primitives, core, global tokens, think of spacer scale, color palettes
• a layer of semantic tokens that refer to the primitives, this layer is often dynamic through theming, meaning that your background accent color primary may be referring to colors.red.500 in one theme, but colors.blue.500 in another, and there can be multiple sublayers here for example, it may be colors.blue.800 in dark mode but colors.blue.300 in light mode.
• a layer of component or option/semantic tokens that are mostly just pure references to the semantic dynamic layer, meaning that you benefit from outputting references here, so you can keep the CSS exactly the same, and through CSS variable references the rendered values change when the active theme changes.

There's many variations of this but it always tends to come down to a form static -> dynamic -> static and only the final static layer really benefits from using CSS variables, at least that has been my observation looking at a lot of people's tokens over the past year.

This is exactly how our tokens are built. And the last layer is where I want the references. But it does not work because border is a component token.

Within this last layer there are references. From my experience expecting to only have one token is not realistic for even semi-complex systems.

Consider the following, you have a color: borderColor.danger that may be used by itself, but you also want to offer a border.danger which would need to use the borderColor.danger token and some size tokens. This already breaks the system.

I don't think this is a pretty complex or "edge-casey" idea.

Let's say borderColor.danger is red, but you want to offer a user with color vision deficiency to change this color, as they can't differentiate red from green.

Now you would either have to build a complex logic that replaces all uses of a specific hex value in your css, OR you just set the borderColor.danger token (css variable) to whatever the user wants, since this is referenced everywhere else.

Other use cases include adjusting contrast, choosing custom colors for components, etc.

[lukasoppermann]

Hey @jorenbroekema,

I still agree with your argument. We have a lot of cases because we don't export our base tokens. Any composite token, like border or shadow which normally references at least a color does not work.

I now created a work around for borders and would love your feedback. I also have an idea for how we could adjust style dictionary, but I will explain this in the end of this comment.

I had to create custom outputReferencesTransformed function:

import type {Dictionary, TransformedToken} from 'style-dictionary/types'
import {resolveReferences} from 'style-dictionary/utils'

export function outputReferencesTransformed(
  token: TransformedToken,
  {dictionary, usesDtcg}: {dictionary: Dictionary; usesDtcg?: boolean},
): boolean {
  const originalValue = usesDtcg ? token.original.$value : token.original.value
  const value = usesDtcg ? token.$value : token.value

  // double check if this is a string, technically speaking the token could also be an object
  // and pass the usesReferences check
  if (typeof originalValue === 'string') {
    // Check if the token's value is the same as if we were resolve references on the original value
    // This checks whether the token's value has been transformed e.g. transitive transforms.
    // If it has been, that means we should not be outputting refs because this would undo the work of those transforms.
    return (
      value ===
      resolveReferences(originalValue, dictionary.unfilteredTokens ?? dictionary.tokens, {
        usesDtcg,
        warnImmediately: false,
      })
    )
  }
  if (typeof originalValue === 'object') {
    const originalValues = Object.values(originalValue).filter(val => typeof val === 'string')

    return originalValues.every(origVal => {
      const resolvedValue = resolveReferences(origVal, dictionary.unfilteredTokens ?? dictionary.tokens, {
        usesDtcg,
        warnImmediately: false,
      })

      return typeof resolvedValue === 'string' ? value.split(' ').includes(resolvedValue) : true
    })
  }
  return false
}

As you can see, I am checking objects as well to make sure they get passed into the process if they have unchanged references.

The other thing I had to adjust was the createPropertyFormatter function.

First I added this part to the value transformation:

if (!originalIsObject) {
  value = originalValue;
} else {
  // new code from here
  if (token.$type === "border") {
    const transformedValues = value.split(" ");
    value = ["width", "style", "color"]
      .map((prop, index) => {
        if (originalValue[prop].startsWith("{")) {
          return originalValue[prop];
        }
        return transformedValues[index];
      })
      .join(" ");
  }
}

Then I also adjusted the value part in the refs.forEach loop:

refs.forEach(ref => {
  if (Object.hasOwn(ref, `${usesDtcg ? '$' : ''}value`) && Object.hasOwn(ref, 'name')) {
    const refVal = usesDtcg ? ref.$value : ref.value
    const replaceFunc = function () {
      //...
    }
    // technically speaking a reference can be made to a number or boolean token, in this case we stringify it first
    value = `${value}`.replace(
      (value as string).match(/{/) ? new RegExp(`{${ref.path.join('\\.')}(\\.\\$?value)?}`, 'g') : refVal,
      replaceFunc,
    )
  }
})

My thinking is, why couldn't we add my change or a similar one to the outputReferencesTransformed function so that the objects are not excluded (they get excluded in the createPropertyFormatter anyway). The only issue I see is that we have to define separators. I am using " " for css, but this does not always work. But maybe we have to life with users having to write a custom outputReferencesTransformed or we could add a outputReferencesTransformedCss function.

And then we allow users to add they own function to deal with objects that is loaded into the createPropertyFormatter in the else section. Users can do whatever they want, but by default nothing happens and the behavior does not change. If they change it, it is their own responsibility, so you don't have to deal with edge-cases.

Of course, we would need to add the small change to the refs.forEach loop, but it does not change anything for the default case either.

[lukasoppermann]

@jorenbroekema There is actually another change needed:

I need to create a custom outputReferencesFilterObject function.

1. I can't import GroupMessages though, any tips?
2. I need to change from every to some, as it is a possible case that one reference was transformed, or has to be transformed, and another not. Does that make sense?
3. I am jumping through so many files, that I am a bit afraid that I am missing something. I think with this code I always add the reference, however, the idea is to only add it, if I actually have it in the output. The problem I am seeing though is that I have multiple calls to create outputs. Sizes, including border-width-default are created in a separate call. But this would mean they are not part of the source section and thus should actually not end up in the output, right? So to make this work I need to load all tokens at the same time into one run of style dictionary, right?

// const FILTER_WARNINGS = GroupMessages.GROUP.FilteredOutputReferences

import type {Dictionary, TransformedToken} from 'style-dictionary/types'
import {getReferences} from 'style-dictionary/utils'

/**
 * @typedef {import('../../../types/DesignToken.d.ts').TransformedToken} TransformedToken
 * @typedef {import('../../../types/DesignToken.d.ts').Dictionary} Dictionary
 *
 * @param {TransformedToken} token
 * @param {{ dictionary: Dictionary, usesDtcg?: boolean }} dictionary
 * @returns
 */
export function outputReferencesFilterObject(
  token: TransformedToken,
  {dictionary, usesDtcg}: {dictionary: Dictionary; usesDtcg?: boolean},
): boolean {
  const originalValue = usesDtcg ? token.original.$value : token.original.value
  // get refs, pass unfilteredTokens to ensure we find the refs even if they are filtered out
  const refs = getReferences(originalValue, dictionary.tokens, {
    unfilteredTokens: dictionary.unfilteredTokens,
    usesDtcg,
    warnImmediately: false,
  })

  return refs.some(ref => {
    // check whether every ref can be found in the filtered set of tokens
    const foundToken = dictionary.allTokens.find(thisToken => thisToken.name === ref.name)
    if (!foundToken) {
      // remove the warning about this ref being filtered out, since we now prevent it from outputting it as a ref
      // GroupMessages.remove(FILTER_WARNINGS, ref.path.join('.'))
    }

    return !!foundToken
  })
}

[lukasoppermann]

@jorenbroekema I have another question:

This part in the createPropertyFormatter.js:

          value = `${value}`.replace(
            originalIsObject ? refVal : new RegExp(`{${ref.path.join('\\.')}(\\.\\$?value)?}`, 'g'),
            replaceFunc,
          );

Why do you do originalIsObject ? refVal. This will replace the real values like 2px with references in an object, right? At least that happens for me. But this can lead to issues were you get references for things that are not in the output e.g. tokens from includes.

I now "solved" this by removing the condition:

          const regex = new RegExp(`{${ref.path.join('\\.')}(\\.\\$?value)?}`, 'g')
          value = `${value}`.replace(regex, replaceFunc)

And above I added some logic to only replace values with references if they are in the source for objects.

} else {
        if (token.$type === 'border') {
          const transformedValues = value.split(' ')
          value = ['width', 'style', 'color']
            .map((prop, index) => {
              if (
                originalValue[prop].startsWith('{') &&
                refs.find(ref => ref.path.join('.') === originalValue[prop].replace(/[{}]/g, ''))?.isSource === true
              ) {
                return originalValue[prop]
              }
              return transformedValues[index]
            })
            .join(' ')
        }
      }

Does this make sense?

[jorenbroekema]

Alright so I understand wanting to go in this direction but there are still use cases that will break:

• If the stringified object value has duplicates, e.g.

  { foo: '{something}', bar: '{somethingelse}' }

  Imagine that this coincidentally ends up resolving to: { foo: '5px', bar: '5px' } which gets transformed to string 5px 5px
  Which 5px is which now? How do we know which value should be var(--something) and which one var(--somethingelse)
• If the stringified object value has duplicates after transformation. Same as the use case above, but it's not the original values, but rather the transformed values that happen to end up as the same. E.g. 4px and 5px but foo properties get increased by +1 they both end up as 5px and cause the duplicates issue
• Similar to the use cases above, imagine a similar example except it ends up resolving to { foo: '10px', bar: '5px', qux: '5px' } which gets transformed to string 10px 5px. So the transformation filtered out the bar or qux property but we do not know which. In fact, maybe foo was reduced by 5 and bar was increased by 5 and we end up accidentally flipping the refs.

I will admit that these are edge cases but not quite so uncommon that I'm comfortable with your changes. I can maybe maybe get behind enabling them as an opt-in if the user feels comfortable and is okay with the risk... After all, if they know a particular token suffers from one or more of these edge case problems, they can manually disable the outputReferences for that particular token.

To explain the originalIsObject.. We're using this to check if the original value of the token was of type object, and if it is, we want to outputReferences in the transformed value rather than doing it in the original value. Doing the regex replaces on the original value is something we started doing in V4 to fix bugs like these: #974 but it's only possible for tokens that were originally not of type object (so strings, numbers etc.).
We can't do a regex replace on an object, so we'll have to assume that the transformed value, which is a string, can be used for that. That said, the transformed value also has its references resolved, so we have to fallback to the "v3 way of doing things" which means regex replacing refs on the transformed/resolved value-parts rather than the original value's references (v4 way, less bug prone).
There is actually a bug with this approach, similar to that linked issue:

{
  offsetX: '5px',
  offsetY: '{foo}',
  blur: '5px',
  color: '#000',
}

Which transforms to 5px 5px 5px #000.
It will outputRefs and do var(--foo) 5px 5px #000 because it has no idea about which property maps to which string-part.
The current implementation of outputReferencesTransformed prevents this bug from happening by just not outputting refs.

Lastly, the outputReferencesFilter is using every because if a value has 2 references and only 1 is in the output, it still should not output references for that value. Right now, outputReferences isn't set up to output some refs and not others, it's either true or false for a token. I'm open to allowing outputReferences functions returning an allowlist Array of refs and make it a bit more flexible in that regard, but we should discuss this in a separate issue imo.

So yeah conclusion imo is that we cannot safely go with your approach, even though the use cases are edge cases, I've seen them happen a lot (especially the shadow example since it has many "dimension" properties that often have the same value).
It may be technically possible to solve these problems with object-string transformation maps so we know which string-part maps to which object property, but I would rather go with thinking outside of the box a bit rethinking the lifecycle:

• certain transforms should apply before reference resolution, e.g. object-to-string, so { foo: '{something}', bar: '{else}' } can become '{something} {else}'. These must be transforms that only reorder/change data structure, no changing the value itself. We will store this value as "intermediateValue" or something like that, similar to "originalValue"
• then all of the regular transforms happen which also includes resolving refs first (or transitively)
• any extra deferred transforms can happen, see Defer / control the order of transforms, e.g. after reference resolution #1063
• when we get to outputReferences in format lifecycle, we can use the "intermediateValue" which will be '{something} {else}', and then check if that's the same as the "value" (after regular transforms) when the refs are resolved, if so, it's safe to outputReferences and because the intermediateValue has the refs in the exact locations, it'll always be safe to do it.

This will impact the user API for the transform lifecycle since there will now be 3 levels of transforms for value transforms:

• pre-resolve -> refs will not be resolved yet, so only use for reordering/changing data structure -> store in intermediateValue (exact property name t.b.d.)
• regular resolve (+ transitive) -> the transform lifecycle as we know it right now, will resolve references and transitive transforms can be applied here, of which the order is defined by the reference chain depth (for transitive ones)
• post-resolve -> here you can apply final transformations that will always be ordered after the steps before this, whereas with transitive transforms you cannot control the order. Main use case: you transitively transformed math expressions or color modifiers, and now want to change the dimension unit/color syntax of the computed value

But imo this really does solve a lot of the issues by tackling this on a slightly higher level. We just have to figure out what the right names are and how to make this API as intuitive as possible, because it is pretty complex.

All of this is the next big priority for Style Dictionary after the performance epic is done #1427 , that one also will contain minor breaking changes and probably a v5 to be published, and I think this topic will also trigger a major bump... so I also have to talk to @dbanksdesign about how we're going to roll these things out with release candidates etc.

For now, my advice for the time being is still to restructure your tokens so that the tokens for which you require outputting refs, are merely "dumb" tokens that are ONLY a reference to another token:

• primitives
• semantics (these may be object values etc., change by active theme)
• others (usually component tokens) -> these may only be "dumb" tokens or direct references, that way you guarantee that outputting references for them is always possible