Skip to content

log: Update log related metrics with a certian interval #11084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
+69 −0
Closed

log: Update log related metrics with a certian interval #11084

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
57507e3
log: Update log related metrics with a certain interval
cosmo0920 Oct 30, 2025
ca9dc71
log: Address a coderabitai's comment
cosmo0920 Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions include/fluent-bit/flb_log.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ struct flb_log {
uint16_t level; /* level */
char *out; /* FLB_LOG_FILE or FLB_LOG_SOCKET */
pthread_t tid; /* thread ID */
uint64_t next_hb_ns; /* next heartbeat (nano sec) */
uint64_t hb_interval_ns; /* heartbeat interval (nano sec) */
struct flb_worker *worker; /* non-real worker reference */
struct mk_event_loop *evl;
struct flb_log_metrics *metrics;
Expand Down
67 changes: 67 additions & 0 deletions src/flb_log.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,70 @@ static inline const char *flb_log_message_type_str(int type)
}
}

#ifdef WIN32
static uint64_t monotonic_now_ns(void)
{
static LARGE_INTEGER s_freq = {0};
LARGE_INTEGER cnt, freq;
uint64_t q = 0;
uint64_t f = 0;
uint64_t rem = 0;
uint64_t sec = 0;

if (s_freq.QuadPart == 0) {
QueryPerformanceFrequency(&s_freq);
}
freq = s_freq;
QueryPerformanceCounter(&cnt);

/* integer math: ns = cnt * 1e9 / freq */
/* avoid 128-bit: split to microseconds first if you prefer */
q = (uint64_t)cnt.QuadPart;
f = (uint64_t)freq.QuadPart;
/* Use MulDiv-like scaling to reduce precision loss */
/* Use 128-bit intermediate or chunk the calculation to avoid overflow */
sec = q / f;
rem = q % f;
return (sec * 1000000000ULL) + ((rem * 1000000000ULL) / f);
}

#else
static uint64_t monotonic_now_ns(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
#endif

static void flb_log_metrics_heartbeat(struct flb_log *log, uint64_t wall_now_ns)
{
uint64_t now_mono = 0;
int log_message_type;
char *message_type_str;

if (!log || !log->metrics || !log->metrics->logs_total_counter) {
return;
}

now_mono = monotonic_now_ns();
if (now_mono < log->next_hb_ns) {
return;
}

for (log_message_type = FLB_LOG_ERROR; log_message_type <= FLB_LOG_TRACE; log_message_type++) {
message_type_str = flb_log_message_type_str(log_message_type);
if (!message_type_str) {
break;
}
cmt_counter_add(log->metrics->logs_total_counter,
wall_now_ns,
0, 1, (char*[]){message_type_str});
}

log->next_hb_ns = now_mono + log->hb_interval_ns;
}

int flb_log_set_file(struct flb_config *config, char *out)
{
struct flb_log *log = config->log;
Expand Down Expand Up @@ -577,6 +641,8 @@ struct flb_log *flb_log_create(struct flb_config *config, int type,
log->out = out;
log->evl = evl;
log->tid = 0;
log->hb_interval_ns = 60ULL * 1000000000ULL;
log->next_hb_ns = 0;
Comment on lines +644 to +645
Copy link

@coderabbitai coderabbitai bot Oct 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Hardcoded heartbeat interval should be configurable.

The heartbeat interval is hardcoded to 60 seconds (line 641), but the PR test configuration uses scrape_interval=60s in the fluentbit_metrics input. If users configure a different scrape interval, the hardcoded heartbeat won't align with the expected metric update frequency.

Consider:

  1. Reading the scrape interval from the fluentbit_metrics input configuration
  2. Making the heartbeat interval configurable via a parameter
  3. Documenting why 60 seconds was chosen if it's intentional to be fixed

Initial heartbeat fires immediately.

Setting next_hb_ns = 0 (line 642) means the first heartbeat will trigger on the first log message, since monotonic_now_ns() returns a positive value. This may be intentional to ensure metrics are emitted early, but consider setting next_hb_ns = monotonic_now_ns() + hb_interval_ns if you want the first heartbeat to fire after one full interval.

🤖 Prompt for AI Agents 
In src/flb_log.c around lines 641-642, the heartbeat interval is hardcoded to
60s and next_hb_ns is set to 0; change this so the hb interval is configurable
and the initial heartbeat is scheduled correctly: read the scrape interval from
the fluentbit_metrics input (or expose a new parameter) and use that value (with
a sensible default of 60s) to set log->hb_interval_ns, and set log->next_hb_ns =
monotonic_now_ns() + log->hb_interval_ns so the first heartbeat fires after one
interval (also add a short comment and update docs/config example to note the
new parameter or why 60s is default).

Copy link
Contributor Author

@cosmo0920 cosmo0920 Oct 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a draft proposal so we wouldn't be needed to make configurable for now.
So, we'll choose option 3 for now.

Copy link

@coderabbitai coderabbitai bot Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cosmo0920, understood! Since this is a draft proposal, keeping the hardcoded 60-second interval is perfectly reasonable for now. You can make it configurable later if needed once the approach is validated. Thanks for clarifying! 👍


ret = flb_pipe_create(log->ch_mng);
if (ret == -1) {
Expand Down Expand Up @@ -826,6 +892,7 @@ void flb_log_print(int type, const char *file, int line, const char *fmt, ...)
}

ts = cfl_time_now();
flb_log_metrics_heartbeat(config->log, ts);
ret = cmt_counter_inc(config->log->metrics->logs_total_counter,
ts,
1, (char *[]) {msg_type_str});
Expand Down
Loading