Filter v1alpha2 — fluentbit.fluent.io

aws object

Aws defines a Aws configuration.

accountID boolean

The account ID for current EC2 instance.Default is false.

alias string

Alias for the plugin

amiID boolean

The EC2 instance image id.Default is false.

az boolean

The availability zone; for example, "us-east-1a". Default is true.

ec2InstanceID boolean

The EC2 instance ID.Default is true.

ec2InstanceType boolean

The EC2 instance type.Default is false.

hostName boolean

The hostname for current EC2 instance.Default is false.

imdsVersion string

Specify which version of the instance metadata service to use. Valid values are 'v1' or 'v2'.

enum: v1, v2

privateIP boolean

The EC2 instance private ip.Default is false.

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

vpcID boolean

The VPC ID for current EC2 instance.Default is false.

customPlugin object

CustomPlugin defines a Custom plugin configuration.

config string

Config holds any unsupported plugins classic configurations, if ConfigFileFormat is set to yaml, this filed will be ignored

yamlConfig object

YamlConfig holds the unsupported plugins yaml configurations, it only works when the ConfigFileFormat is yaml

grep object

Grep defines Grep Filter configuration.

alias string

Alias for the plugin

exclude string

Exclude records which field matches the regular expression. Value Format: FIELD REGEX

logicalOp string

Specify the logical operator for multiple regex/exclude rules.

enum: and, or

regex string

Keep records which field matches the regular expression. Value Format: FIELD REGEX

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

kubernetes object

Kubernetes defines Kubernetes Filter configuration.

alias string

Alias for the plugin

annotations boolean

Include Kubernetes resource annotations in the extra metadata.

bufferSize string

Set the buffer size for HTTP client when reading responses from Kubernetes API server.

pattern: ^\d+(k|K|KB|kb|m|M|MB|mb|g|G|GB|gb)?$

cacheUseDockerId boolean

When enabled, metadata will be fetched from K8s when docker_id is changed.

dnsRetries integer

DNS lookup retries N times until the network start working

format: int32

dnsWaitTime integer

DNS lookup interval between network status checks

format: int32

dummyMeta boolean

If set, use dummy-meta data (for test/dev purposes)

k8sLoggingExclude boolean

Allow Kubernetes Pods to exclude their logs from the log processor (read more about it in Kubernetes Annotations section).

k8sLoggingParser boolean

Allow Kubernetes Pods to suggest a pre-defined Parser (read more about it in Kubernetes Annotations section)

keepLog boolean

When Keep_Log is disabled, the log field is removed from the incoming message once it has been successfully merged (Merge_Log must be enabled as well).

kubeCAFile string

CA certificate file

kubeCAPath string

Absolute path to scan for certificate files

kubeMetaCacheTTL string

configurable TTL for K8s cached metadata. By default, it is set to 0 which means TTL for cache entries is disabled and cache entries are evicted at random when capacity is reached. In order to enable this option, you should set the number to a time interval. For example, set this value to 60 or 60s and cache entries which have been created more than 60s will be evicted.

kubeMetaNamespaceCacheTTL integer

Configurable TTL for K8s cached namespace metadata. By default, it is set to 900 which means a 15min TTL for namespace cache entries. Setting this to 0 will mean entries are evicted at random once the cache is full.

format: int32

kubeMetaPreloadCacheDir string

If set, Kubernetes meta-data can be cached/pre-loaded from files in JSON format in this directory, named as namespace-pod.meta

kubeTagPrefix string

When the source records comes from Tail input plugin, this option allows to specify what's the prefix used in Tail configuration.

kubeTokenCommand string

Command to get Kubernetes authorization token. By default, it will be NULL and we will use token file to get token.

kubeTokenFile string

Token file

kubeTokenTTL string

configurable 'time to live' for the K8s token. By default, it is set to 600 seconds. After this time, the token is reloaded from Kube_Token_File or the Kube_Token_Command.

kubeURL string

API Server end-point

kubeletHost string

kubelet host using for HTTP request, this only works when Use_Kubelet set to On.

kubeletPort integer

kubelet port using for HTTP request, this only works when useKubelet is set to On.

format: int32

labels boolean

Include Kubernetes resource labels in the extra metadata.

mergeLog boolean

When enabled, it checks if the log field content is a JSON string map, if so, it append the map fields as part of the log structure.

mergeLogKey string

When Merge_Log is enabled, the filter tries to assume the log field from the incoming message is a JSON string message and make a structured representation of it at the same level of the log field in the map. Now if Merge_Log_Key is set (a string name), all the new structured fields taken from the original log content are inserted under the new key.

mergeLogTrim boolean

When Merge_Log is enabled, trim (remove possible \n or \r) field values.

mergeParser string

Optional parser name to specify how to parse the data contained in the log key. Recommended use is for developers or testing only.

namespaceAnnotations boolean

Include Kubernetes namespace resource annotations in the extra metadata.

namespaceLabels boolean

Include Kubernetes namespace resource labels in the extra metadata.

namespaceMetadataOnly boolean

Include Kubernetes namespace metadata only and no pod metadata. If this is set, the values of Labels and Annotations are ignored.

ownerReferences boolean

Include Kubernetes owner references in the extra metadata.

regexParser string

Set an alternative Parser to process record Tag and extract pod_name, namespace_name, container_name and docker_id. The parser must be registered in a parsers file (refer to parser filter-kube-test as an example).

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

tlsDebug integer

Debug level between 0 (nothing) and 4 (every detail).

format: int32

tlsVerify boolean

When enabled, turns on certificate validation when connecting to the Kubernetes API server.

useJournal boolean

When enabled, the filter reads logs coming in Journald format.

useKubelet boolean

This is an optional feature flag to get metadata information from kubelet instead of calling Kube Server API to enhance the log. This could mitigate the Kube API heavy traffic issue for large cluster.

useTagForMeta boolean

If true, Kubernetes metadata (e.g., pod_name, container_name, namespace_name etc) will be extracted from the tag itself.

logToMetrics object

LogToMetrics defines a Log to Metrics Filter configuration.

addLabel []string

Add a custom label NAME and set the value to the value of KEY

alias string

Alias for the plugin

bucket []string

Defines a bucket for histogram

discardLogs boolean

Flag that defines if logs should be discarded after processing. This applies for all logs, no matter if they have emitted metrics or not.

emitterMemBufLimit string

set a buffer limit to restrict memory usage of metrics emitter

emitterName string

Name of the emitter (advanced users)

exclude []string

Optional filter for records in which the content of KEY does not matches the regular expression. Value Format: FIELD REGEX

kubernetesMode boolean

If enabled, it will automatically put pod_id, pod_name, namespace_name, docker_id and container_name into the metric as labels. This option is intended to be used in combination with the kubernetes filter plugin.

labelField []string

Includes a record field as label dimension in the metric.

metricDescription string

Sets a help text for the metric.

metricMode string

Defines the mode for the metric. Valid values are [counter, gauge or histogram]

metricName string

Sets the name of the metric.

metricNamespace string

Namespace of the metric

metricSubsystem string

Sets a sub-system for the metric.

regex []string

Optional filter for records in which the content of KEY matches the regular expression. Value Format: FIELD REGEX

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

tag string

Defines the tag for the generated metrics record

valueField string

Specify the record field that holds a numerical value

lua object

Lua defines Lua Filter configuration.

alias string

Alias for the plugin

call string required

Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the Script defined above.

code string

Inline LUA code instead of loading from a path via script.

enable_flb_null boolean

If enabled, null will be converted to flb_null in Lua. This helps prevent removing key/value since nil is a special value to remove key/value from map in Lua. Default value: false.

protectedMode boolean

If enabled, Lua script will be executed in protected mode. It prevents to crash when invalid Lua script is executed. Default is true.

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

script object

Path to the Lua script that will be used.

key string required

The key to select.

name string

Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

optional boolean

Specify whether the ConfigMap or its key must be defined

timeAsTable boolean

By default when the Lua script is invoked, the record timestamp is passed as a Floating number which might lead to loss precision when the data is converted back. If you desire timestamp precision enabling this option will pass the timestamp as a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.

typeArrayKey []string

If these keys are matched, the fields are handled as array. If more than one key, delimit by space. It is useful the array can be empty.

typeIntKey []string

If these keys are matched, the fields are converted to integer. If more than one key, delimit by space. Note that starting from Fluent Bit v1.6 integer data types are preserved and not converted to double as in previous versions.

modify object

Modify defines Modify Filter configuration.

alias string

Alias for the plugin

conditions []object

All conditions have to be true for the rules to be applied.

aKeyMatches string

Is true if a key matches regex KEY

keyDoesNotExist object

Is true if KEY does not exist

keyExists string

Is true if KEY exists

keyValueDoesNotEqual object

Is true if KEY exists and its value is not VALUE

keyValueDoesNotMatch object

Is true if key KEY exists and its value does not match VALUE

keyValueEquals object

Is true if KEY exists and its value is VALUE

keyValueMatches object

Is true if key KEY exists and its value matches VALUE

matchingKeysDoNotHaveMatchingValues object

Is true if all keys matching KEY have values that do not match VALUE

matchingKeysHaveMatchingValues object

Is true if all keys matching KEY have values that match VALUE

noKeyMatches string

Is true if no key matches regex KEY

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

rules []object

Rules are applied in the order they appear, with each rule operating on the result of the previous rule.

add object

Add a key/value pair with key KEY and value VALUE if KEY does not exist

copy object

Copy a key/value pair with key KEY to COPIED_KEY if KEY exists AND COPIED_KEY does not exist

hardCopy object

Copy a key/value pair with key KEY to COPIED_KEY if KEY exists. If COPIED_KEY already exists, this field is overwritten

hardRename object

Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists. If RENAMED_KEY already exists, this field is overwritten

remove string

Remove a key/value pair with key KEY if it exists

removeRegex string

Remove all key/value pairs with key matching regexp KEY

removeWildcard string

Remove all key/value pairs with key matching wildcard KEY

rename object

Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists AND RENAMED_KEY does not exist

set object

Add a key/value pair with key KEY and value VALUE. If KEY already exists, this field is overwritten

multiline object

Multiline defines a Multiline configuration.

alias string

Alias for the plugin

buffer boolean

emitterMemBufLimit integer

Set a limit on the amount of memory in MB the emitter can consume if the outputs provide backpressure. The default for this limit is 10M. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to 10MB then the pipeline will pause if the buffer exceeds 10M. The pipeline will remain paused until the output drains the buffer below the 10M limit.

emitterName string

Name for the emitter input instance which re-emits the completed records at the beginning of the pipeline.

emitterType string

The storage type for the emitter input instance. This option supports the values memory (default) and filesystem.

enum: memory, filesystem

flushMs integer

keyContent string

Key name that holds the content to process. Note that a Multiline Parser definition can already specify the key_content to use, but this option allows to overwrite that value for the purpose of the filter.

mode string

enum: parser, partial_message

parser string required

Specify one or multiple Multiline Parsing definitions to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma.

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

nest object

Nest defines Nest Filter configuration.

addPrefix string

Prefix affected keys with this string

alias string

Alias for the plugin

nestUnder string

Nest records matching the Wildcard under this key

nestedUnder string

Lift records nested under the Nested_under key

operation string

Select the operation nest or lift

enum: nest, lift

removePrefix string

Remove prefix from affected keys if it matches this string

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

wildcard []string

Nest records which field matches the wildcard

parser object

Parser defines Parser Filter configuration.

alias string

Alias for the plugin

keyName string

Specify field name in record to parse.

parser string

Specify the parser name to interpret the field. Multiple Parser entries are allowed (split by comma).

preserveKey boolean

Keep original Key_Name field in the parsed result. If false, the field will be removed.

reserveData boolean

Keep all other original fields in the parsed result. If false, all other original fields will be removed.

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

unescapeKey boolean

If the key is a escaped string (e.g: stringify JSON), unescape the string before to apply the parser.

recordModifier object

RecordModifier defines Record Modifier Filter configuration.

alias string

Alias for the plugin

allowlistKeys []string

If the key is not matched, that field is removed.

records []string

Append fields. This parameter needs key and value pair.

removeKeys []string

If the key is matched, that field is removed.

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

uuidKeys []string

If set, the plugin appends uuid to each record. The value assigned becomes the key in the map.

whitelistKeys []string

An alias of allowlistKeys for backwards compatibility.

rewriteTag object

RewriteTag defines a RewriteTag configuration.

alias string

Alias for the plugin

emitterMemBufLimit string

emitterName string

When the filter emits a record under the new Tag, there is an internal emitter plugin that takes care of the job. Since this emitter expose metrics as any other component of the pipeline, you can use this property to configure an optional name for it.

emitterStorageType string

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

rules []string

Defines the matching criteria and the format of the Tag for the matching record. The Rule format have four components: KEY REGEX NEW_TAG KEEP.

throttle object

Throttle defines a Throttle configuration.

alias string

Alias for the plugin

interval string

Interval is the time interval expressed in "sleep" format. e.g. 3s, 1.5m, 0.5h, etc.

pattern: ^\d+(\.[0-9]{0,2})?(s|m|h|d)?$

printStatus boolean

PrintStatus represents whether to print status messages with current rate and the limits to information logs.

rate integer

Rate is the amount of messages for the time.

format: int64

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

window integer

Window is the amount of intervals to calculate average over.

format: int64

wasm object

Wasm defines a Wasm configuration.

accessiblePaths []string

Specify the whitelist of paths to be able to access paths from WASM programs.

alias string

Alias for the plugin

eventFormat string

Define event format to interact with Wasm programs: msgpack or json. Default: json

functionName string

Wasm function name that will be triggered to do filtering. It's assumed that the function is built inside the Wasm program specified above.

retryLimit string

RetryLimit describes how many times fluent-bit should retry to send data to a specific output. If set to false fluent-bit will try indefinitely. If set to any integer N>0 it will try at most N+1 times. Leading zeros are not allowed (values such as 007, 0150, 01 do not work). If this property is not defined fluent-bit will use the default value: 1.

pattern: ^(((f|F)alse)|(no_limits)|(no_retries)|([1-9]+[0-9]*))$

wasmHeapSize string

Size of the heap size of Wasm execution. Review unit sizes for allowed values.

pattern: ^\d+(k|K|KB|kb|m|M|MB|mb|g|G|GB|gb)?$

wasmPath string

Path to the built Wasm program that will be used. This can be a relative path against the main configuration file.

wasmStackSize string

Size of the stack size of Wasm execution. Review unit sizes for allowed values.

pattern: ^\d+(k|K|KB|kb|m|M|MB|mb|g|G|GB|gb)?$