LLM powered next word/sentence auto-suggestions (like in smartphone keyboards)?

[life00]

Recently I was wondering if it would be possible to implement word/sentence auto-suggestions similar to how smartphone software keyboards handle it but as completion popup or ghost text. I tried implementing something closely related to this using cmp-dictionary and cmp-spell, but its not very effective because it does not consider the context sentence.

After I discovered llm.nvim I thought maybe its still possible? I tried to make it work but it seems that the LSP was primarily made for code suggestion. So I wanted to know if what I am suggesting is possible with this tool? Its not necessarily a feature request, but I was just wondering.

Ideally it would be nice if the implementation could run a small model like gemma:2b (it is necessary to tell gemma to only output text, otherwise it adds unnecessary comments) and only read current line context to improve performance (I believe currently the whole file is inputted?). In addition to that it would be nice to have it as nvim-cmp source (for word suggestions), and ghost text for sentence suggestions.

[McPatate]

You can tune your configuration settings such as context_window (you can lower this to something like 256 or even lower to make requests faster) and in request_body, if you are using ollama you can try this:

  request_body = {
    options = {
      num_predict = 10, -- you can lower this if you want
    },
  },

Let me know if this makes things better. Note that reducing context_window will impact the quality of suggestions.

Btw there has been some interest in creating an nvim-cmp plugin, but haven't gotten the time to make it.