Sensor base 0.76.6#146
Closed
djoreilly wants to merge 2021 commits into
Closed
Conversation
…ds (#4616) For our env the connection timeout is only 15sec and for splunk hecs it's 12sec under pressure (https://docs.splunk.com/Documentation/Splunk/8.2.6/admin/Inputsconf#http:_.28HTTP_Event_Collector.29 - busyKeepAliveIdleTimeout), but most of the cases it should be over 60..
IMO helpful for testing in forks
I’ve just seen #4577 and #4587 — this one is probably on me.
In #3270 I incorrectly assumed that .Default("A") would take effect, but
it looks like it didn’t because the field may have been an empty string
(e.g. same for server_rpm_command_output -
https://github.com/Velocidex/velociraptor/blob/master/bin/rpm.go#L140-L143).
Sorry for the oversight on my part.
Simple documentation fix that updates the reference to `oidc-cognito` in `server.config.yaml` to specify that the authenticator only exists prior to v0.75.6. It was removed in favor of the regular `oidc` authenticator as the issues that required a separate authenticator were fixed in the AWS upstream repository. This pull request addresses #4622.
* Images can be opened in a larger modal dialog * External links must open in a new tab
Using a policy allows the kernel to pre-filter events before passing them to the query, thereby reducing CPU load.
The event will only be distributed to the watcher that matched the policy. Also updated the ebpf tracker to report on each lister and policy.
…(#4631) This causes deadlocks on systems with only one core.
Correct typo within the interactive configuration generation.
This pull request addresses the issue identified in #4636. The return value of this function unintentionally mutated the `components` slice passed in which led to unexpected behavior when handling OSPath instances across multiple calls. This was resolved by cloning the `components` slice before modification to ensure that the original slice remains unmodified.
Fix command and generalize link
…entConfig (#4630) Velociraptor installs its configuration in `%ProgramFiles%/Velociraptor/client.config.yaml`. This change expands the path using the environment in `Admin.Client.UpdateClientConfig` to ensure that a configuration change can be applied across hosts with non-standard `%ProgramFiles%` directory locations.
…… (#4646) Add version to Windows.Detection.Amcache to allow hutning on version
The new process tracker is more efficient: 1. Process Children are stored in the tracked process entry avoiding the need to sweep across the entire tracker when constructing the process tree 2. The tracker now uses links to maintain correspondence between process ID (which is heavily reused) and internal unique ID (based on the start time). This avoids the need to sweep through the tracker when we detect a PID was reused. 3. The tracker can use an on disk sqlite based LRU which is a bit slower but reduces memory use and allows tracking much larger number of processes.
Update Windows.Forensics.Amcache to replace Windows.System.Amcache This update adds additional AMCache types as referenced by recent research: - https://www.ssi.gouv.fr/uploads/2019/01/anssi-coriin_2019-analysis_amcache.pdf - https://github.com/libyal/dtformats/blob/main/documentation/AMCache%20file%20(AMCache.hve)%20format.asciidoc Contributed by InfoGuard IR team
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Also: * Upload Function/Plugin Minimum Filesize Bug * Page the notebook overview table when there are many cells in the notebook. Fixes: #4643 #4642 --------- Co-authored-by: snyk-bot <snyk-bot@snyk.io>
…r hunts. (#4615) Added logic to map clients by hostname if the Client ID is unknown for hunts. This is related to: #4614 --------- Co-authored-by: Mike Cohen <mike@velocidex.com>
We now use the same code for forwarding logs to syslog: 1. Syslog now uses octet counting transport 2. Syslog log forwarding now works on Windows as well.
Sometimes when the server crashes or due to various reasons, the hunt stats become out of step with the actual data. This shows up as a 0 in scheduled clients, or the numbers are not quite right. This PR performs occasional sweeps over the hunts to fix up the stats from the flows themselves. This allows us to keep high level stats like: - Errored clients - Total bytes/rows transferred - Scheduled clients etc.
If flows crashed before sending any status then the server would not update them. This resulted in repeatedly sending flow status request without making progress.
Add regex filters for IPs and ports in KernelNetwork.yaml
This pull request fixes an issue where the current reformat logic could
unintentionally overwrite content following the `query` field in an
artifact. This most commonly affected artifacts that define additional
fields such as `notebook` or `column_types`.
For example, given the following artifact with a `notebook` field:
```yaml
name: Notebook
sources:
- query: |
SELECT A,B,C
FROM scope()
notebook:
- name: Test
type: vql_suggestion
template: |
SELECT * FROM scope()
```
Running the reformat command (either via CLI or the GUI) would remove
the `notebook:` line. Subsequent reformats would continue stripping
content, producing an invalid artifact such as:
```yaml
name: Notebook
sources:
- query: |
SELECT A,
B,
C
FROM scope()
- name: Test
type: vql_suggestion
template: |
SELECT * FROM scope()
```
> Error: While parsing VQL at line 8: unexpected token "-"
The root cause was an off-by-one error in the mutation logic applied to
the `query` block, which caused the reformat operation to extend beyond
the intended range. While addressing this, I also identified an issue
with how `original_end_line` was calculated for chomping indicator nodes
(`|-`). Because these nodes omit the trailing newline, splitting on `\n`
resulted in an incorrect end line calculation.
Additionally, the `current_mu.err` value was never checked when applying
mutations that only had a length of one. This has also been resolved by
adding an additional check at the beginning of the line iteration.
This change corrects these issues and adds regression tests to verify
that:
- Content following a `query` block is preserved.
- Running reformat multiple times produces stable, idempotent output.
Please let me know if you'd like further clarification, or if there are
any changes you'd like made.
I am also considering a refactor of the `artifacts reformat` to call a
`reformat` VQL function which wraps this functionality, following the
pattern that CLI commands are just wrappers around VQL. Let me know if
this is also something you'd be interested in me contributing. 😃
Profiling with pprof/allocs showed newAuditBuf() to be allocating more than expected. Also there are a copylock problems: go vet vql/linux/audit/*go vql/linux/audit/audit_service.go:177:18: literal copies lock value from rawBufPool: sync.Pool contains sync.noCopy go vet utils/refcount.go utils/refcount.go:15:9: return copies lock value: command-line-arguments.Refcount contains sync/atomic.Int32 contains sync/atomic.noCopy After fixing the copylocks by using pointers, trying to reuse buffers from the pool would cause a panic because the refcount was zero. Fix by resetting the refcount before returning the buffer the pool.
Profiling with pprof/allocs shows LeaseSerialized() in file_store/directory/buffer.go to be the top offender allocating 4096 byte slices every time, most of which never get used.
The read_file function without a length argument was allocating 4MB buffers and adding a lot of work for the garbage collector. Also it often didn't work as the parent process would be gone by the time the lookup was done, leading to the client log getting spammed with "no such file or directory" messages. Since we gather all proceeses and the PPIDs it's possible to find the parent cmdline anyway, so it can be removed here to fix these problems.
The ipv6 connect probes were not attached.
For ipv6 the remote address was being reported for the local address
and vice versa. Also renamed the saddr_v* and daddr_v* variables as
they don't make sense in the outbound case.
From comment in sock.h:
skc_daddr: Foreign IPv4 addr
skc_rcv_saddr: Bound local IPv4 addr
skc_v6_daddr: IPV6 destination address
skc_v6_rcv_saddr: IPV6 source address
To simplify and be consistent with all other plugins which leave the hashing to VQL in the artifact. Also the VQL hash function has caching and our other artifacts also give the SHA1.
Deferred calls were used to close files from a loop, but these won't run until the function returns. Move the whole event parsing into its own function and use os.Stat() instead. Also there is no need to convert the timestamp to a string.
Some errors logs where missing context making it difficult to know which plugin or where in the plugin the error came from.
Also there is no need to convert time to a string or have json tags.
When the client event table changes, Velociraptor cancels each plugin's context and expects it to shut itself down. Then it starts them again if they are still needed in the new table. The bpf based plugins were not stopping and Velociraptor was starting new goroutines for them, leaving the old ones running and leaking resources forever. This commit changes the bpf plugins so they now exit when the context is cancelled. Velociraptor does not wait until the plugins are fully stopped before starting new instances and the concurrent loading and unloading of BPF modules is problematic. To ensure only one BPF module load or unload operation is happening at any one time, this commit introduces a publish/subscribe service.
This plugin uses eBPF and has less performance overhead than using the audit subsystem to capture execve system calls. Also it does not drop events under heavy load conditions. It requires linux 5.8+ for eBPF ringbuffer support - artifacts should check that before calling it. It uses tracepoint/sched/sched_process_exec because tracepoint/syscalls/sys_enter_execve is not reliable bpftrace/bpftrace#2523 (comment)
The execsnoop plugin can be used if it's in the binary and the kernel is 5.8+ required for eBPF ringbuffer support. Otherwise the audit plugin will be used as before.
If more than one instance of the audit plugin is called with watch rules for the same path but each using different keys, e.g.: SELECT * FROM audit(rules=["-w /etc/passwd -k key1"]) WHERE "key1" IN Tags SELECT * FROM audit(rules=["-w /etc/passwd -k key2"]) WHERE "key2" IN Tags Then it turns out that each event will only be sent to one of the instances. Fix by using the same key for the same paths across instances. The audit plugin reference counts audit rules and duplicates are not a problem.
Update our fork of go-rpmdb to fix a panic when modernc.org/libc tries to read a badly formatted /etc/services: https://gitlab.com/cznic/libc/-/issues/25
This artifact sends events from configured client event artifacts to
the local rsyslog listening with the imuxsock input module on the
specified unix domain socket.
The plugin uses a file backed ring buffer when needed and should be able
to survive short outages of the rsyslog daemon (like restarts) without
losing events - up to 1GB of events.
Tested with this rsyslog config:
input(type="imuxsock" socket="/tmp/velo-socket" ruleset="velo-ruleset")
ruleset(name="velo-ruleset") {
action(type="omfile" file="/tmp/velo.log")
}
The SHA256 is sufficient for Treat Intelligence identification. Also name the column consistenly as "SHA256".
Updates each event query to include the machine-id for SENS-149. A materialized call is used to read /etc/machine-id, so it's only done once per query.
Sometimes grok() fails to parse the output of systemctl and the logs get spammed with "symbol not found in scope" error messages. Fix by not referencing the expected symbols in the serviceDetails() function as there is no need. They are referenced from a dict in the main SELECT and get the value of null if they don't exist without any error messages.
This reduces the binary size by ~15MB.
Prior to v0.74 the VQL array() function would flatten a subquery. This is documented and some of our artifacts depended on this behaviour. https://github.com/Velocidex/velociraptor/blob/v0.7.0/docs/references/vql.yaml#L110 In v0.74 this was changed and now a different syntax is needed to get the same Velocidex/velociraptor@7652599 Our artifacts need to work with a mix of clients versions, some have the old behaviour and some have the new, and they all get the same artifacts from the server. Unfortunately the version of array() was not bumped with this breaking change, so version() can't be used in an if() statement. This fix uses python with the starl() plugin instead of array(), as a common syntax that works with all versions could not be found.
We were parsing the output of `systemctl show` with grok(), but the order of properties in the output appears to be random and can change with systemd version. Fix by simply adding the key/value pairs to a dict.
Our original version depended on the upstream Linux.Ssh.AuthorizedKeys artifact, but it changed significantly and ours no longer works with it. Fix by copy/pasting the parts that parse the authorized_keys files. Having no dependency is less brittle in the long term.
This also updates github.com/ProtonMail/gopenpgp
Our build rpm spec runs a sed command to change the VERSION constant to a format semver can't parse and the build fails. As we don't need these offline dependencies we can just skip creating them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase commits from sensor-base-0.7.0 onto upstream v0.76.6