@rwalkerands : thank you for opening up this issue! 🤗

How would you expect to separate out multiple output streams if --template and --json both go to standard output?

I don't really see a clean way to achieve what you're hoping for unfortunately. That said, I can talk about how I typically work with JSON for scripting without duplicate gh calls.

Like mona-actions/gh-repo-stats extension, you can dump the JSON to a file on disk and work with it as base-64 strings allowing you to loop over entries with normal shell scripting:

  REPOS=$(echo "${PARSE_DATA}" | jq -r '.data.organization.repositories.nodes')
  for REPO_DATA in $(echo -n "${REPOS}" | jq -r '.[] | @base64'); do
    _jq() {
    echo -n "${REPO_DATA}" | base64 --decode | jq -r "${1}"
    }

    OWNER=$(_jq '.owner.login' | tr '[:upper:]' '[:lower:]')
    REPO_NAME=$(_jq '.name' | tr '[:upper:]' '[:lower:]')

This way, you can pull and even paginate data while limiting the number of calls you're trying to make.

I followed this issue because I had the same feature request, and am surprised this is considered too difficult to implement.

The template feature is just wrapping templating functionality in Go to my understanding, and the feature request is that the input to this template can be filtered using jq.

At least, in my case that was all I wanted, and was surprised to find it incompatible (so I had to pipe to jq manually, despite this being bundled in gh, making my script less portable).

I don't need to handle "multiple output streams" and I don't want to rely on writing to disk, this seems pretty simple: just pipe the output of the jq filter into the template format.

How would you expect to separate out multiple output streams if --template and --json both go to standard output?

I wouldn't, which is why I offered two proposed solutions which are different from that.

To repeat them:

1. Add an option to output the JSON to a file (i.e., with the filename specified as the parameter of the option).
2. Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

To expand on (2): add a new top-level command, say gh template --template <template>, that takes JSON input either from standard input or from a file specified as another command-line option (or, I guess the JSON could also be provided as a command-line argument), and applies the template provided as the command-line argument. (By "JSON input" I mean, of course, the output of a previous gh ... -jq ... command.)

I happen to like the implementation of templates provided by gh. Here's part of my script that produces a report on open PRs for a repo:

fields=assignees,author,baseRefName,body,createdAt,headRefName,headRepositoryOwner,id,labels,number,reviewDecision,reviews,statusCheckRollup,title,updatedAt,url
#PR=$(gh pr list -R "${REPO}" --json "${fields}"  --jq '.[]')

gh pr list -R "${REPO}" --json "${fields}" \
--template \
'====================================================================================================
{{range .}}
{{printf "#%v" .number | color "green"}} {{.title}}
{{.url}}
Author: {{.author.name}} (@{{.author.login}} https://github.com/{{.author.login}})
Created at: {{timeago .createdAt}}
Updated at: {{timeago .updatedAt}}
{{if .reviewDecision}}Review decision: {{.reviewDecision}}{{end -}}

This is a request to merge: {{.headRepositoryOwner.login}}/{{.headRefName}} -> {{.baseRefName}}

Description:

{{.body}}

{{if .labels}}{{tablerow "LABEL" "DESCRIPTION"}}{{range .labels}}{{tablerow .name .description}}{{end}}{{tablerender}}
{{end -}}
{{if .assignees}}{{tablerow "ASSIGNEE" "NAME"}}{{range .assignees}}{{tablerow .login .name}}{{end}}{{tablerender}}
{{end -}}
{{if .reviews}}{{tablerow "REVIEWER" "STATE" "COMMENT"}}{{range .reviews}}{{tablerow .author.login .state .body}}{{end}}{{tablerender}}
{{end -}}
{{if .statusCheckRollup}}{{tablerow "CHECK" "START_TIME" "STATE" "URL"}}{{range .statusCheckRollup}}{{tablerow .context .startedAt .state .targetUrl}}{{end}}{{tablerender}}
{{end -}}
====================================================================================================
{{end}}

'

Yes, I probably could implement this template using Bash/jq. But I'd rather not have to.

@rwalkerands : Thanks for clarifying your thoughts further as initially it seemed you might be asking for something that was impossible: having JSON and template output both streaming through standard out. Seeing that you and @lmmx were considering multi-phase processing via JSON + JQ then template processing, I believe this issue has come up previously as I recall asking a similar question.

At the time, the feedback I received was:

--jq and --template were both designed to convert JSON into something more convenient for terminals. I guess you could in theory use --jq to massage data passed to --template (if we allowed the combination of flags) but I think allowing that would be fiddly and not lead to great user experiences. it would be better to add helpers to --template to make massaging data there more easier

Personally, I'm open to the idea still though the --jq support isn't 1:1 to jq; see https://github.com/itchyny/gojq#difference-to-jq for more information. Understanding how this will play out with @heaths changes in #1268 for being able to provide a single merged JSON response from the GitHub API will be interesting.

Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

This is certainly outside of the scope of what the GitHub CLI team and maintainers can support. It is something I myself have looked into for similar reasons however I didn't truly find anything that allowed users to arbitrary define Go templates and pass in JSON via file or standard input.

Add an option to output the JSON to a file (i.e., with the filename specified as the parameter of the option).

Currently, this wouldn't work or work well given commands that require a --paginate flag or have to iterate on multiple pages of data from the API.

Why not just have gh emit JSON, tee that, and send stdout through jq or some Go templating app. Off hand I'm not aware of one, but you could use a similar design to my extension https://github.com/heaths/gh-merge-json (gh ext install heaths/gh-merge-json) that is an alternative to #1268 until I was allowed to complete that work (soon).

E.g.,

gh api ... | tee output.json | jq ...

Having a CLI do "everything" seems not only a maintenance issue, but also limiting: it's not always going to work the way people want, and as a consequence devs keep having to add knobs which increases maintenance costs for complexity. Keeping it flexible might be better.

If you want something easy to use in scripts, consider at least keeping it in the ecosystem as an extension. You could easily have an extension both tee and format JSON using Go templates and install that if needed with the --force switch I added back in #7173 for that very purpose that I use it for in many of our scripts:

#!/usr/bin/env bash

# this will install or uprade the extension, or no-op if already installed
gh ext install --force heaths/gh-merge-json

# using `gh ext exec` is good for scripts and helps avoid future possible ambiguity, which happened to me a while back
gh api ... | gh ext exec merge-json

Just my $0.02.

I added support to merge-json in heaths/gh-merge-json#6 and pushed a new release. Seems appropriate, given that the merge-json extension requires JSON input anyway...for which I also fixed an input validation bug prior to this PR.

Example usage:

gh ext install --force heaths/gh-merge-json > /dev/null # or just $null in pwsh
gh api graphql --paginate -f query='
    query($endCursor: String) {
      viewer {
        repositories(first: 100, after: $endCursor) {
          nodes { nameWithOwner }
          pageInfo {
            hasNextPage
            endCursor
          }
        }
      }
    }
  ' | gh merge-json --template '{{range .data.viewer.repositories.nodes}}{{tablerow .nameWithOwner}}{{end}}' --tee repos.json

Why not just have gh emit JSON, tee that, and send stdout through jq ...

Because using jq to implement the sort of template(s) I want to have (see #8415 (comment)) appears to be quite cumbersome ... if it is even possible. (E.g., how to do {{timeago .createdAt}}? An inability to do that wouldn't be a showstopper, but it's nice to be able to mimic what's shown in the GitHub web interface.)

or some Go templating app. Off hand I'm not aware of one,

You answered your own question. I already looked, but didn't find. Very happy to be pointed at a small standalone-app that does JSON through a Go-like template to text.

Indeed, thank you for the pointer to https://github.com/heaths/gh-merge-json ; I have now tried it and confirmed that it works for me.

Thanks for sharing your insights, @heaths ❤️ There is definitely a fine line of how GitHub CLI aligns with other tools in the Unix design principle we're trying to exercise being an extremely small team.

All of that to say, @rwalkerands , I hope you have some options for short/medium term workarounds as discussed above while I bring this up for discussion with the other maintainers on ramifications of refactoring the code involved:

All of that to say, @rwalkerands, I hope you have some options for short/medium term workarounds as discussed above while I bring this up for discussion with the other maintainers ...

Indeed, I am now using gh merge-json --template and it does what I need.

[shameless plug] @rwalkerands if you like it, consider starring heaths/gh-merge-json to to help discovery on https://github.com/topics/gh-extension 😉 If not, that's cool too. Won't change support or feature ideas unlike some repos in the news recently. Just helps others find it easier in case it might help them.

On that note: @andyfeller it'd be cool if GitHub defaulted a search from a topic to include that topic when you hit / or click in the search box e.g. topic:gh-extension merge json. I use search a lot and didn't even realize topic: was a valid prefix. Just guessed it might work and tried it with pleasant surprise!

consider starring heaths/gh-merge-json to

Done (and was happy to do so).

On that note: @andyfeller it'd be cool if GitHub defaulted a search from a topic to include that topic when you hit / or click in the search box e.g. topic:gh-extension merge json. I use search a lot and didn't even realize topic: was a valid prefix. Just guessed it might work and tried it with pleasant surprise!

Kind of like jumping to search within a repo will add repo:..., owner:..., or repo:... path:....

At the time, the feedback I received was:

--jq and --template were both designed to convert JSON into something more convenient for terminals. I guess you could in theory use --jq to massage data passed to --template (if we allowed the combination of flags) but I think allowing that would be fiddly and not lead to great user experiences. it would be better to add helpers to --template to make massaging data there more easier

I recently raised this, which was closed as a dupe of this issue:

• Allow --jq and --template to be used together in gh api #10261

I'd like to add my support for having whatever data manipulations I provide in --jq passed through to --template as a 'pipeline'.

It might be fiddly, but that's something I as a user should be allowed to choose to opt into IMO. There shouldn't be any backwards compatibility issues as --jq + --template is currently disallowed.

Personally I find the fact that I can't use both together a worse user experience.

I have no strong opinions on whether --template could also be improved to make massaging data there easier.. but IMO that should be a separate enhancement; that would then give users more choice as to whether they want to use the fiddly --jq + --template, or if they want to use whatever new features may eventually end up in --template.

tl;dr: Supporting --jq + --template feels like a quick/easy win, whereas changes to --template feels like a larger scope with more potential for delays/scope creep/etc.

Because using jq to implement the sort of template(s) I want to have (see #8415 (comment)) appears to be quite cumbersome ... if it is even possible. (E.g., how to do {{timeago .createdAt}}?

@rwalkerands Since I hacked something together for this recently, this was how I solved it:

• https://github.com/0xdevalias/dotfiles/blob/843359eb91e2530b27d27baccb5a08f707bd1a63/bin/gh-notifications#L75-L87

def timeago:
  . | fromdateiso8601 | now - . | . / 60 | floor |
  if . < 60 then
    (. | tostring + \" minutes ago\")
  elif . < 1440 then
    (. / 60 | floor | tostring + \" hours ago\")
  elif . < 43200 then
    (. / 1440 | floor | tostring + \" days ago\")
  elif . < 525600 then
    (. / 43200 | floor | tostring + \" months ago\")
  else
    (. / 525600 | floor | tostring + \" years ago\")
  end;

I also raised this, which might be helpful if possible/implemented

• Add custom formatting functions to --jq similar to those added to --template #10262

Most commands use github.com/cli/cli/cmdutil.AddJSONFlags or .../cmdutil.AddFormatFlags, which do not enforce --jq and --template to be mutually exclusive:

gh issue list --json number,title --jq 'sort_by(.title)' --template '{{range .}}{{.title}}{{end}}

However, in this example since --jq is processed first, it writes directly to stdout. Given that, supporting --jq processing before --template could be considered a breaking change; but, I still think it's a good idea if you want to do additional processing.

gh api does treat --jq and --template as mutually exclusive, though. With a little refactoring it could be possible to serialize processing --jq then --template.

@andyfeller after an initial attempt to do this entirely in cli/cli, while it was pretty straight forward - though it felt a little tacky - in the cmdutil package, the api command would've felt even more hacky given where the template was initialized and how even more disjoint the input and output buffers were across functions.

With that in mind, and because I think jq filters offer a lot more flexibility for extension developers than Go's simplistic (in mostly good ways) template engine, I created cli/go-gh#177 with a very simple change to use the existing jq package and is much less code churn than in cli/cli (see the first commit for this PR, which I left to give you some idea; imagine worse for the api command).

@heaths @0xdevalias we see the value of this, and it doesn't seem particularly tricky to implement based on the proposed PRs. That said, we are a bit swamped at the moment with other work so it may take a little time to ensure we are confident in the UX and documentation that needs to go with it.

@heaths do you see value for extension authors putting this in go-gh, or was it mainly a matter of ease/simplicity in the solution?

Both. It definitely keeps the code in the CLI cleaner, but also allows extension authors to easily filter JSON output before applying templates (baked in or otherwise).

E.g., say you wanted to output the top 10 issues by vote in a nice, neat table. If your GraphQL statement had something like:

reactions(content: THUMBS_UP) {
  totalCount
}

You can easily morph and filter:

t := template.New(ios.Out, ios.Width(), ios.ColorEnabled());
t.Parse("{{range .}}{{tablerow .number .title .votes}}{{end}}")
template.ExecuteFiltered(r,
    "map({number, title, votes: .reactions.totalCount}) | sort_by(.votes) | reverse | .[:2]")

Much easier than all the Go code to do the same.