A curated collection of diagnostic and administrative tools for Splunk environments, maintained by Splunk Support for customer use. These scripts help troubleshoot common issues, validate configurations, and perform routine maintenance tasks.
- Splunk Support Scripts
KV Store Certificate Verifier - Verify KV Store certificates before MongoDB upgrade
- Problem: Need to verify KV Store certificates before upgrading MongoDB from 4/4.2 to 7
- Solution: Validates SSL settings, certificate chains, and compatibility requirements
Splunk Config Checker - Generic configuration validator
- Problem: Need to validate Splunk configurations across multiple conf files
- Solution: JSON-based rule validator with custom severity levels and detailed reporting
Splunk User Permissions Checker - Audit user permissions and conflicts
- Problem: Need to audit user permissions and identify conflicts
- Solution: Queries REST API to show role assignments, capabilities, index access, and permission conflicts
Find Duplicate Inputs - Detect overlapping monitor inputs
- Problem: Duplicate data ingestion from overlapping monitor inputs
- Solution: Scans inputs.conf files to identify duplicate or overlapping monitor paths
Lookup Generator - Generate large test lookup files
- Problem: Need large test lookup files for performance testing
- Solution: Generates CSV files with randomized data up to specified size (default 14MB)
Test Peers - Verify distributed search peer connectivity
- Problem: Need to verify distributed search peer connectivity
- Solution: Auto-discovers peers and tests TCP connectivity on port 8089
Debugging Splunk - Stack Collection Tool - Advanced pstack collection for main splunkd (Linux only)
- Problem: Need detailed user-space stack traces from splunkd for debugging lock contention, deadlocks, and thread issues
- Solution: Advanced collection tool with multiple modes (batch, continuous, Docker), sophisticated sampling, and optional process freezing
Kernel Stacks - D-State Monitor - Monitor D-state processes and collect kernel stacks (Linux only)
- Problem: Need to diagnose processes stuck in uninterruptible sleep (D-state) waiting for I/O or resources
- Solution: Continuously monitors and captures kernel stacks specifically for D-state processes to identify I/O bottlenecks
Kernel Stacks Splunk Threads - Monitor Splunk threads and collect kernel stacks (Linux only)
- Problem: Need to diagnose Splunk thread exhaustion and capture system state during incidents
- Solution: Monitors splunkd thread count and automatically collects kernel stack traces when threshold exceeded
Splunkd Pstacks Threads Monitor - Monitor main splunkd threads and collect pstacks (Linux only)
- Problem: Need to automatically capture user-space stack traces when splunkd thread count exceeds 500
- Solution: Monitors main splunkd process and collects pstack dumps to diagnose lock contention and thread exhaustion
Process-Runner Pstacks Threads Monitor - Monitor process-runner threads and collect pstacks (Linux only)
- Problem: Need to diagnose thread issues specific to Splunk process-runner (scripted inputs, custom commands)
- Solution: Monitors process-runner thread count and collects pstack dumps to identify issues with external process execution
New Log Event - Generate Windows event log entries (Windows only)
- Problem: Need to test Windows event log ingestion with specific event sizes
- Solution: PowerShell script to generate custom-sized event log entries
Most scripts require one or more of:
- Splunk Enterprise installation
- Splunk's Python interpreter (
$SPLUNK_HOME/bin/python) - Bash shell (Linux/macOS)
- PowerShell 3.0+ (Windows-specific tools)
Refer to individual tool documentation for specific requirements.
Each tool has its own README with detailed usage instructions, parameters, and troubleshooting guidance. Navigate to the tool's directory and review the README.md file.
Team members can contribute new scripts by following these guidelines:
- Create a dedicated folder using kebab-case naming (e.g.,
my-new-tool/) - Include the script(s) with appropriate execute permissions for shell scripts
- Create a README.md with the following sections:
- Purpose (what problem it solves)
- Requirements
- Usage (with clear examples)
- Parameters/options
- Example output (if helpful)
- Notes (compatibility, limitations, etc.)
- Update top-level README.md to add your tool to the appropriate category
# Tool Name
Brief description of what the tool does.
## Purpose
Explain the problem this tool solves and use cases.
## Requirements
- List dependencies
- Note OS compatibility
- Specify version requirements
## Usage
\`\`\`bash
./script.sh [options]
\`\`\`
### Options
- `-o, --option` - Description
### Examples
\`\`\`bash
# Example 1
./script.sh --example
# Example 2
./script.sh --another-example
\`\`\`
## Notes
- Important compatibility notes
- Known limitations
- Security considerations (if applicable)- Scripts should include error handling
- Shell scripts must have shebang lines (
#!/bin/bash) - Cross-platform compatibility when possible (test on Linux/macOS)
- Clear comments for complex logic
- Security best practices (avoid hardcoded credentials, validate inputs)
- Scripts cannot change customer environment apart from creating logging files
- When using python, if at all possible, use the python packaged with Splunk for consistencies sake
Send scripts to the repository maintainer for review. Include:
- Script file(s)
- README documentation
- Brief description of testing performed
- Tyler Ezell (tezell@splunk.com) - Initial scripts and maintenance
- Rob Hilgefort (rhilgefort@splunk.com) - Repository setup, review, and stewardship
- Robert Phillips (rphillips@splunk.com) - Performance diagnostics and stack collection tools
Added:
debugging-splunk- Advanced pstack collection tool for main splunkd with multiple collection modeskernel-stacks- D-state process monitor for diagnosing uninterruptible sleep issuessplunkd-pstacks-threads- Automatic pstack collection when main splunkd thread count exceeds 500process-runner-pstacks-threads- Automatic pstack collection when process-runner thread count exceeds 500
Included:
kvcertverify- KV Store certificate verifier for MongoDB upgradessplunk_config_checker- Generic configuration validatorsplunk-user-permissions- User permissions and conflicts auditorfind-duplicate-inputs- Duplicate monitor input detectorlookup-generator- Large test lookup file generatortest-peers- Distributed search peer connectivity testerkernel-stacks-splunk-threads- Kernel stack collector triggered by Splunk thread thresholdnew-log-event- Windows event log entry generator