The native logging service provider implementation of elf4j (Easy Logging Facade for Java), and an independent drop-in logging solution for any Java application
As an application developer using the elf4j logging facade, I want to have the option of using a runtime log service provider that natively implements the API and SPI of elf4j.
- Java 8+ required for versions earlier than 14.0.0 (exclusive)
- Java 21+ required for versions later than 14.0.0 (inclusive)
-
Guiding principle: Reasonable default and Pareto's 80/20 rule
-
This is simply a packaging unit of the
elf4j-engine, using the Java Service Provider Framework mechanism. For more implementation detail, see the elf4j-engine code repo.
<dependency>
<groupId>io.github.elf4j</groupId>
<artifactId>elf4j-provider</artifactId>
<version>${latest.version}</version>
</dependency>elf4j-provider output is always asynchronous. This is for throughout performance and, moreover, the 80/20 rule: When was the last time a use case truly required that logging had to be synchronous, and always blocking the application's normal work flow?
Only standard streams (stdout/stderr) channels are supported as log output destinations. It has
8000
been common, yet arguably redundant, that an application logging framework supports other output channels - given how specially and comprehensively the application's hosting system is already equipped to redirect/forward standard-stream data to other destinations. The elf4j-provider/elf4j-engine does not consider such data collecting process as an application-level concern - be it as simple as a Linux shell redirect to (rotating) log files, or as sophisticated as running a collector agent of a comprehensive observability service (e.g. Splunk, ELK, Newrelic, Datadog, etc...).
JSON is a supported output pattern, in hopes of helping external log analysis tools. This is in addition to the usual line-based patterns - timestamp, level, thread, logger, class, method, file name, line number, and log message.
-
Supports configuration refresh during runtime via, with option of passing in replacement properties instead of reloading the configuration file. The most frequent use case would be to change the minimum log output threshold level, without restarting the application.
-
To avoid loss of logs when the application shuts down, it is the user's responsibility to call
LogServiceManager.stopbefore the application exits. Upon that call, the log service will- stop accepting new log events
- block and wait for all the accepted log events to finish processing
Alternatively, the user can register a JVM shutdown hook provided by
LogServiceManager.getShutdownHookThread.
-
As with any other elf4j logging provider, client application should code against service API of the elf4j facade, and drop in this provider implementation as a runtime dependency shown in the "Installation" section.
-
See elf4j for API sample usage.
-
In case of multiple elf4j service providers in classpath, pick this one like so:
java -Delf4j.service.provider.fqcn="elf4j.engine.NativeLoggerFactory" MyApplicationMore details here.
The default configuration file location is at the root of the application class path, with file
name elf4j-test.properties, or if that is missing, elf4j.properties. Alternatively, to override the default
location, use Java system property to provide an absolute path:
java -Delf4j.properties.location=/absoloute/path/to/myConfigurationFile -jar MyApplicaiton.jar
Absence of a configuration file results in no logging (no-op) at runtime. When present, though, the configuration file
requires zero/no configuration thus can be empty: the default configuration is a stdout writer with a minimum level
of TRACE and a basic line-based logging pattern. To customize the default logging configuration, see the
configuration sample file below.
A log output is rendered only when the Logger instance severity level is on or above the minimum output levels for
both the log caller class and the log writer.
- The default severity level of a
Loggerinstance fromLogger.instance()isINFO, which is not configurable: The elf4j API should be used to programmatically obtainLoggerinstances of desired severity levels. - The default minimum output level for both log caller classes and the writer and is
TRACE, which is configurable: For caller classes, the minimum output level can be configured on global, package, or individual class levels.
By default, the elf4j-engine supports one single writer instance of the standard-stream type. The codebase is extension-ready for multiple writer types; and for each custom type, multiple writer instances. However, the need for such extensions (e.g. a flat-file writer type with multiple writer instances for different target files) is rare, considering the abundant host/OS and vendor options to ship standard-stream data to various destinations other than the default system console.
All individual patterns, including the JSON pattern, can either be the only output of the log entry, or mixed together
with any other patterns. They each take the form of {pattern:displayOptions}, where multiple display options are
separated by commas. Patterns inside curly brace pairs are predefined and will be interpreted before output, while
patterns outside curly brace pairs are written out verbatim.
The predefined patterns are:
timestamp: Date time format configurable via JavaDateTimeFormatterpattern syntax, default to ISO datetime format with time zone offset of the application running hostlogger: Option ofsimple,full, orcompressed(only the first letter for each package segment) for class names, default tosimplelevel: Length configurable, default to full lengththread: Option ofnameorid, default to nameclass(performance-sensitive): Option ofsimple,full, orcompressed(only the first letter for each package segment) for class names, default tosimplemethod(performance-sensitive): No configuration options, simple method namefilename(performance-sensitive): No configuration options, simple file namelinenumber(performance-sensitive): No configuration options, the line number where the log is issued in the filesysprop: Optionnamefor the JRE System Propertysysenv: Optionnamefor the system environment variablemessage: No configuration options, always prints user message, and exception stack trace if anyjson: Multiple options allowed -caller-threadto include caller thread (name and id),caller-detailto include caller stack detail (class, method, filename, linenumber), and optionprettyto indent the JSON text to more readable format. Default is no thread/caller detail and the minified single-line formatcontext: Mandatory display optionkeywhose value will be retrieved from theMDCcontext and displayed in the log message; thekeycannot be empty, otherwise, an error will be raised
Note:
- Pattern names are case-insensitive.
- A system-dependent line feed character (hidden pattern) is appended automatically at the end of each log entry; this is not configurable.
- Performance-sensitive patterns should only be used when necessary or if the application is performance tolerant.
Line-based Default
-
Pattern: none, which is the same as
{timestamp} {level} {logger} - {message} -
Output:
2023-03-22T21:11:33.040-05:00 INFO IntegrationTest$defaultLogger - Hello, world!
Line-based Customized
- Pattern:
{timestamp:yyyy-MM-dd'T'HH:mm:ss.SSSXXX} {level:5} [{thread:name}] {class:compressed}#{method}(L{linenumber}@{filename}) -- {message} - Output:
2023-03-30T17:14:54.735-05:00 INFO [main] e.e.IntegrationTest$defaultLogger#hey(L50@IntegrationTest.java) -- Hello, world!
JSON Default
- Pattern:
{json} - Output:
{"timestamp":"2023-10-07T22:52:47.962872-05:00","message":"Hello, world!","level":"INFO","callerClass":"elf4j.provider.Demo"}
JSON Customized
- Pattern:
{json:caller-thread,caller-detail,pretty} - Output:
{ "timestamp": "2023-10-07T22:53:46.2568507-05:00", "message": "Hello, world!", "level": "INFO", "callerThread": { "name": "main", "id": 1 }, "callerDetail": { "className": "elf4j.provider.Demo", "methodName": "main", "lineNumber": 12, "fileName": "Main.java" } }
### Zero configuration mandatory, this file can be empty - default to a line-based writer with simple log pattern
### global no-op flag, overriding and will turn off all logging if set true
#noop=true
### Global minimum output level for both caller class and writer. Optional, default to TRACE.
level=info
### These override the minimum output level of all caller classes included the specified package spaces
level@org.springframework=warn
level@org.apache=error
### Standard out stream type, stdout or stderr, default is stdout
stream=stderr
### pattern is optional, default is a simple line-based
pattern={json}
#pattern={json:caller-thread,caller-detail,pretty}
#pattern={timestamp:yyyy-MM-dd'T'HH:mm:ss.SSSXXX} {level:5} [{thread:id}] {class:compressed}#{method}(L{linenumber}@{filename}) - {message}
### Max concurrency to process logs from different caller threads, default to available runtime processors
#concurrency=20Either stdout (the default if omitted) or stderr, configured globally.
LogServiceManager.refresh() will reload the configuration file and apply the latest file properties during
runtime. LogServiceManager.refresh(java.util.Properties) will apply the passed-in properties as the replacement of the
current properties, and the configuration file will be ignored.
It's not how fast you fill up the target log file or repository, it's how fast you relieve the application from logging duty back to its own business.
Performance benchmark metrics tend to be highly dependent on the nature of the work load and overall setup, but here is a naive start (Recommend re-configuration based on individual use cases), comparing with some popular log engines:
Chronological order is generally required for log events to arrive at their final destination. The usual destination of the standard out streams is the system Console, where an out-of-order display would be confusing. That means log events need to be processed sequentially - at least for those events that are issued from the same application/caller thread. This inevitably imposes some limit on the log processing throughput. No matter the log processing is synchronous or asynchronous to the main business workflow, if the application's log issuing frequency is higher than the throughput of the log processing, then over time, the main workflow should be blocked and bound ("back-pressured") by the log processing throughput limit.
Some logging information has to be collected by the main application thread, synchronously to the business workflow. For example, caller detail information such as method name, line number, or file name are performance-wise expensive to retrieve, yet unavailable for a different/asynchronous thread to look up. The elf4j-engine uses the main caller thread to synchronously collect all required information into a log event, and then hands off the event to an asynchronous process for the rest of the work. It helps, however, if the client application can do without performance-sensitive information in the first place; the default log pattern configuration does not include such caller details.
The ideal situation of asynchronous logging is for the main application to "fire and forget" the log events, and continue its main business flow without further blocking, in which case the log processing throughput has little impact on the main application. That only happens, however, when there are spare threads (and/or task queue capacity) available from the async thread pool. When no spare thread is available, the log process becomes pseudo-synchronous, in which case not only will the main application be back-pressured while awaiting processing time, but also the extra cost of facilitating asynchronous communications will now add to that of the main workflow. By contrast, a true synchronous logging without buffering will delay the main workflow in each transaction, albeit having no additional cost for asynchrony.
For asynchronous logging to work well, the log processing throughput should, over time, exceed the log event generation rate; the work queue hosting the log events should serve only as a temporary buffer when the log eventing rate is momentarily higher than the log processing throughput.
Leveraging the conseq4j concurrent API, the elf4j-engine processes log events issued by different caller threads in parallel, and those by the same caller threads in sequence. This ensures all logs of the same caller thread arrives at the log destination in chronological order (same order as they are issued by the thread). However, logs from different caller threads are not guaranteed of any particular order of arrival.
If omitted in configuration file, the default concurrency (maximum number of threads in parallel) for asynchronous processing is the number of Runtime#availableProcessors of the current JVM at the application startup time (or when the log service is refreshed). This is the thread pool capacity for log event processing.
The standard stream destinations are often redirected/replaced by the host environment or user, in which case further manoeuvres may help such data channel's performance. For example, if the target repository is a log file on disk, then
java MyApplication | cat >logFilemay out 7DD3 perform
java MyApplication >logFiledue to the buffering effect of piping and cat.
Such external setups fall into the category of increasing data channel bandwidth, and are considered outside the scope of application-level logging.