By scroll I mean the cursor doesn't move while the pane itself scrolls up/down

What if the cursor is scrolled to the edge of the page? In that case, doesn't the cursor have to be moved anyway?

By scroll I mean the cursor doesn't move while the pane itself scrolls up/down

What if the cursor is scrolled to the edge of the page? In that case, doesn't the cursor have to be moved anyway?

Yes exactly, the expected behavior is similar to vim's ctrl+e and ctrl+y.
And if scrolloff is set to a value more than 0 then the cursor should start moving according to the scrolloff value.

Something to mention though, it would be nice if the scroll unit is in number of cells and not in terms of the available window space.

I wrote a plugin for it, which basically moves the cursor to the edge of the list to make the list scroll, and then restores the cursor to its original position.

Save it as ~/.config/yazi/plugins/list-seek.yazi/init.lua:

return {
  entry = function(_, args)
    local folder = cx.active.current
    local total = #folder.files - 1
    if total <= 0 then
      return
    end

    local step = tonumber(args[1])
    local start = folder.window[1].idx - 1 + MANAGER.scrolloff
    local end_ = folder.window[#folder.window].idx - 1 - MANAGER.scrolloff

    local bound
    if step > 0 then
      bound = { math.min(total, start + step), math.min(total, end_ + step) }
    else
      bound = { math.max(0, start + step), math.max(0, end_ + step) }
    end

    local target = step > 0 and bound[2] or bound[1]
    local delta = target - folder.cursor
    for _ = 1, math.abs(delta) do
      ya.manager_emit("arrow", { delta > 0 and 1 or -1 })
    end

    ya.manager_emit("arrow", {
      ya.clamp(bound[1], folder.cursor, bound[2]) - target,
    })
  end,
}

And bind these keys to it:

{ on = "<C-j>", run = 'plugin --sync list-seek --args="5"' },
{ on = "<C-k>", run = 'plugin --sync list-seek --args="-5"' },

This plugin also handles scrolloff well, this is what it looks like when I set scrolloff = 0: