Difference in these codes for lua logging http-log/splunk logging?

[jeremyjpj0916]

Originally tried to make my splunk log plugin and http-log plugin use some optimized code yall added and tweak it to work with the splunk:

/services/collector

Base path which requires a more splunk formatted payload than pure json see below example.

This requires the batched tx payload to look like this format which isn't true json:

{log}
{log}
{log}
...

True JSON(how kong does the json batch logging by default):

[
{log},
{log},
....
]

So why does this work for the format I need for the splunk logging:

local table_concat = table.concat
local function json_array_concat(entries)
  --return "[" .. table_concat(entries, ",") .. "]" If splunk followed true json format we would use this COMMENTED OUT
    return "" .. table_concat(entries, "\n\n") .. "" -- Break events up by newlining them
end

But not this refactored version of your own new function yall use:

-- Create a function that concatenates multiple JSON objects into a JSON array.
-- This saves us from rendering all entries into one large JSON string.
-- Each invocation of the function returns the next bit of JSON, i.e. the opening
-- bracket, the entries, delimiting commas and the closing bracket.
-- UPDATE: Edited to not have leading/trailing []'s and no commas between entries for splunk format logging.
local function make_splunk_json_array_payload_function(conf, entries)
  if conf.queue.max_batch_size == 1 then
    return #entries[1], entries[1]
  end

  local nentries = #entries

  local content_length = 1
  for i = 1, nentries do
    content_length = content_length + #entries[i] + 1
  end

  local i = 0
  local last = max(2, nentries * 2 + 1)
  return content_length, function()
    i = i + 1

    if i == 1 then
      return '' --Dropped the [

    elseif i < last then
      return i % 2 == 0 and entries[i / 2] or '\n\n' --Dropped the , for newlines

    elseif i == last then
      return '' --Dropped the ]
    end
  end
end

Any lua gurus see the main diff between the working one for me that gets the splunk format I need vs trying to make yalls new function work by stripping the []'s and ,'s ? Yall definitely moved away from the table concat one due to perf improvements in the memory of it all so I would love to move to it as well if I found what was going wrong with the way I tweaked it thats incorrect. Would even be nice if a splunk flag could be added to http-log plugin OSS to swap between native json and splunk's whacky body requirement too without me having to patch drop in the file manually.

[hanshuebner]

Hello @jeremyjpj0916,

it is a little difficult to follow your writeup, but If I understand correctly, you want to use the http-log plugin to send your logs to splunk. Our documentation describes how to do that, does that not work for you?

[jeremyjpj0916]

Correct it does not work for the splunk logging endpoint I want to use. Yalls docs works for the raw splunk endpoint which comes with another set of issues. I gave 2 lua code blocks above and need to understand what I did wrong that makes the second newer optimal kong lua code block not create a payload body with working output like the first block that uses table concat I included does(and used to be how kong used to do their json logging with the http log plugin long ago before it got the newer refactored performant way to produce the payload for multiple events)

[hanshuebner]

What is "yall", please?

[jeremyjpj0916]

Ah haha, "yall" is A southern United States word thrown around frequently. "You guys" is what folks up north say. "Yall" is what folks down south here say.

Yall is short for: You all, like referring to a collective group. In this instance "yalls docs" means "kongs docs"

[jeremyjpj0916]

Closing this... oddly the http-log plugin seems to be working for me with splunks hec collector on 3.9.0, cant fully explain it but I don't have to patch the http-log plugins code with this anymore, the regular json payload yall do seems to work:

My old hack:

local function json_array_concat(entries)
  --return "[" .. table_concat(entries, ",") .. "]" If splunk followed true json format we would use this
    return "" .. table_concat(entries, "\n\n") .. "" -- Break events up by newlining them
end

....


-- Sends the provided entries to the configured plugin host
-- @return true if everything was sent correctly, falsy if error
-- @return error message if there was an error
local function send_entries(conf, entries)
  local content_length, payload
  if conf.queue.max_batch_size == 1 then
    assert(
      #entries == 1,
      "internal error, received more than one entry in queue handler even though max_batch_size is 1"
    )
    content_length = #entries[1]
    payload = entries[1]
  else
    payload = json_array_concat(entries)
    content_length = #payload
  end