Osquery can let you see the state of your computers right now. However, this snapshot means you have to check the table repeatedly if you want to view data over time. Actively watching and diffing tables could be challenging even with automation, especially for short-lived or high-churn processes.
This is where osquery's evented tables can help. Instead of displaying the point-in-time state for your host, osquery's evented tables store the host's historical data. You can configure osquery to capture certain types of information, which will be stored in the relevant
*_events table for later analysis.
In this guide, we’ll go through the concepts, considerations, and best practices for setting up evented tables — focusing on osquery. We’ll also cover basic information about commonly-used evented tables to help you get started.
Osquery does not generate the events itself. Instead, it reads and formats event data generated by various OS components. For example, on Linux, the audit framework generates and broadcasts process and socket event data. Osquery receives the data, converts it into an event row, and buffers them (in the internal rocksdb store) in the
socket_events tables to await querying. The data can then be filtered and transformed via SQL and shipped to a log destination with the scheduled query functionality.
For the purposes of this article, we'll use the term "utility" to mean the underlying OS component that osquery subscribes to for its various evented tables.
This separation between osquery and the utility means that some evented tables rely on configurations for the utility to determine which events will be generated. At the utility level, you can specify what data is captured. At the osquery level, you can specify what information is ingested, presented, and transmitted. For most evented tables, osquery works great out of the box with the utility’s default configurations, but some use cases may require adjusting the utility configuration.
You can tell that an osquery table is evented in two ways:
Capturing event data generates performance overhead from both the utility and osquery. If the utility is configured loosely to generate more events, then the utility performs more operations to generate events and osquery parses and stores more events.
Capturing only the events you need will cut down on the amount of work the host needs to do. For example, when monitoring processes, there may be frequent but low-value processes such as
sed which can be ignored, reducing work for the host. Collecting “good enough” data is key in managing performance impact.
Also, consider the impact of the queries you’re using to collect your event data. Queries using
WHERE clauses will be fairly efficient (and minimize data volume), while many
JOINs or wildcards (%) will use more resources.
Even after considering all of these factors, you may need to give osquery some additional resources. Osquery's watchdog automatically cancels queries if they exceed certain system usage. This can be adjusted with the following flags:
--watchdog_memory_limit changes the maximum memory usage (expressed in MBs).
--watchdog_utilization_limit changes the maximum number of CPU cycles (defined as the
system_time) for more than the time in seconds set by
--watchdog_delay sets the delay in seconds before CPU and memory usage limits will be enforced (60 seconds by default).
Osquery collects data from the utility, formats it into an event row, and stores it in the evented table for querying. Of course, the more events data is collected, the more disk space this occupies. We recommend that you do not rely on osquery as long-term storage for event data. Instead, regularly schedule the data to be sent to an external destination for future analysis. Osquery has built-in options to automatically clean up the data.
The following osquery flags will help you manage the size of osquery's data:
--events_max sets the maximum number of event rows per evented table to store in the buffer before expiring them with a default value of 50,000.
--events_expiry sets the lifetime of event rows in seconds with a default value of 86,000 (24 hours). An event only expires if a query has been against the table after event generation. When combined with scheduled queries, this is a handy way to clean out data automatically. Some osquery practitioners set this to
1 so that it immediately gets cleared out when a scheduled query runs.
--events_optimize=true saves the time that this table was last queried and only returns events after that time (enabled by default). This can be overridden in a one-off query by specifying the
time column in a
You could also consider configuring the utility to ignore extraneous data to minimize resource utilization. Ignoring extraneous data can minimize disk usage at both the utility and osquery levels. The utility will not generate this data in the first place. Or osquery can filter out the extraneous processes in the
SELECT statement. Both will minimize data volume.
There is always risk of data loss. Both osquery and the utilities limit the disk usage and processor usage, which results in data loss during periods of high system load. Osquery will log when it's dropping events due to high load, though this will not detect when events are dropped by the utility. To address the risk of data loss, you could schedule queries to run more frequently or allocate more resources to osquery or the utility. Of course, this comes at a cost and requires tuning to balance risk and reward.
Getting the right setup that balances performance, data volume, and data usefulness for the evented tables requires some trial and error. The best way is to try things out on a progressively larger set of machines. We recommend setting up a canary team on Fleet to test different combinations of configurations.
osquery_schedule table will list all scheduled queries and recent information about their memory usage and execution time. Note that these do not have visibility into the utility. For lower-level visibility, use the OS-native profilers.
osquery_events table tells you which evented tables are turned on (
active column) and the number of events stored (
events column) per table.
osquery_flags table tells you the current set of flags for osquery. You can use this to confirm the desired flags are set correctly.
osquery_schedule table lists all scheduled queries and collects memory and execution time for the latest execution.
--verbose flag will generate more logs with troubleshooting information.
To turn on osquery's eventing system, set the flag
--disable_events=false. Eventing is disabled by default.
Each evented table is turned on by its own flag. For most evented tables, when you turn them on in osquery, osquery will use the default configuration of the utility. The defaults are good enough for most situations.
However, we recommend getting to know the underlying utility to optimize it for your use case. Let's go over the following topics:
FIM refers to the monitoring of key files or filepaths. FIM enables organizations to audit the history of critical resources, detect intrusions, and apply remediations.
On all three OSs, in the osquery configuration, use the
file_paths key to specify the files and directories from which osquery should collect
file_events data. Use the
exclude_paths key to ignore files and directories that generate too much noise. Wildcards are available in these configuration options. On Linux, there is a further
file_accesses option, which specifies the file locations where an "access" event should be recorded in addition to created/modified/deleted.
file_events on for macOS, use the flag
--enable_file_events=true. The corresponding utility is FSEvents.
MacOS also has an
es_process_file_events table that uses the EndpointSecurity API. However, osquery needs Full Disk Access permission, which can be granted manually or via MDM. To use this, use the flags
es_process_file_events records which processes accessed which files, whereas
file_events does not. However,
es_process_file_events will generate more data volume because it captures everything by default. Currently, you can configure EndpointSecurity to ignore certain file paths, but there is no way to configure it to only watch certain filepaths.
Due to the data volume, Fleet suggests using
file_events for macOS, but you can use
file_events on for Linux, use the flag
--enable_file_events=true. The corresponding utility is inotify.
Linux has a
process_file_events table that uses the audit framework. To use this table, use the flags
Fleet recommends using the
process_file_events table since it also includes data for which process accessed which file.
For Windows, use the
--enable_ntfs_event_publisher=true flag to turn on
ntfs_journal_events. The corresponding utility is NTFS Journal.
Read the osquery FIM docs for more information on file integrity monitoring with osquery.
Process auditing refers to recording process executions and network, or socket, connections.
The choice of utility depends on your situation. Here are some factors to consider:
--audit_persist=true flag will set osquery to retry connection to audit logs.
We recommend that you try both and compare results for your use case.
To use the
bpf_socket_events tables, use the flag
--enable_bpf_events=true. See the instructions on auditing using bpf for more information.
socket_events with the audit framework, use the flags
--disable_audit=false --audit_allow_process_events=true --audit_allow_socket_events=true. See the instructions on using audit for more information.
On macOS, there are two utilities that enable osquery process auditing: OpenBSM and the EndpointSecurity. Fleet recommends using the EndpointSecurity implementation because it's intended to replace OpenBSM, which is deprecated. EndpointSecurity is available starting macOS 10.15.
To use the
es_process_events tables, use the flag
--disable_endpointsecurity=false. See the EndpointSecurity instructions for more information. To use
socket_events with OpenBSM, see the OpenBSM instructions.
Currently, osquery does not support process auditing for Windows. To learn more about process auditing on Windows, visit Microsoft's security auditing overview. Fleet is tracking work to build process auditing for Windows in osquery. Stay up to date on GitHub.
YARA is a malware research and detection tool available on Linux and macOS that allows users to create descriptions of malware families based on patterns of text or binary code. Each potential piece of malware is matched against a YARA rule and triggers if the specified conditions are met.
Osquery applies pre-specified YARA rules to incoming events in the
file_events table to populate the
yara_events table. As such, it requires the following flags:
With the appropriate flags set, specify the appropriate YARA rules in the osquery configuration using the format described in the YARA configuration doc.
Explore more topics and useful links for YARA:
These event tables are available in osquery. We will provide more information for them in other guides.
|no additional flags needed
|no additional flags needed
|no additional flags needed