Learn Splunk in 10 DaysDay 9: Splunk Administration
books.chapter 9Learn Splunk in 10 Days

Day 9: Splunk Administration

What You Will Learn Today

  • User and role management
  • Index management
  • Knowledge objects
  • Splunk apps
  • Configuration file precedence

User and Role Management

Role Hierarchy

flowchart TB
    Admin["admin<br>Full access"]
    Power["power<br>Advanced searches & reports"]
    User["user<br>Basic searches & dashboards"]
    CanDelete["can_delete<br>Event deletion rights"]
    Admin --> Power --> User
    Admin --> CanDelete
    style Admin fill:#ef4444,color:#fff
    style Power fill:#f59e0b,color:#fff
    style User fill:#22c55e,color:#fff
    style CanDelete fill:#8b5cf6,color:#fff

Default Roles

Role Permissions
admin Full control: settings, user management, all data
power Create reports, alerts, and shared dashboards
user Run searches and create personal dashboards
can_delete Delete events from indexes

Creating Users

Navigate to Settings > Users > Add new

Setting Description
Username Login ID
Full name Display name
Email Email address
Roles Assigned roles
Default app The app shown after login

Creating Custom Roles

Navigate to Settings > Roles > Add new

Name: analyst
Inheritance: user
Allowed indexes: main, web_logs, security
Search filter: index=main OR index=web_logs OR index=security
Max search jobs: 5
Max concurrent searches: 3

authorize.conf

# authorize.conf
[role_analyst]
importRoles = user
srchFilter = index=main OR index=web_logs OR index=security
srchIndexesAllowed = main;web_logs;security
srchIndexesDefault = main
srchJobsQuota = 5
cumulativeSrchJobsQuota = 10

Index Management

Designing Your Index Strategy

flowchart TB
    subgraph Indexes["Example Index Layout"]
        Main["main<br>General logs"]
        Web["web_logs<br>Web access logs"]
        Security["security<br>Security events"]
        App["application<br>Application logs"]
        Internal["_internal<br>Splunk internal logs"]
    end
    style Main fill:#3b82f6,color:#fff
    style Web fill:#22c55e,color:#fff
    style Security fill:#ef4444,color:#fff
    style App fill:#f59e0b,color:#fff
    style Internal fill:#8b5cf6,color:#fff

Creating an Index

# indexes.conf
[web_logs]
homePath = $SPLUNK_DB/web_logs/db
coldPath = $SPLUNK_DB/web_logs/colddb
thawedPath = $SPLUNK_DB/web_logs/thaweddb
maxTotalDataSizeMB = 50000
frozenTimePeriodInSecs = 7776000
maxDataSize = auto_high_volume

Retention Settings

Parameter Description Example
frozenTimePeriodInSecs Data retention period 7776000 (90 days)
maxTotalDataSizeMB Maximum data size 50000 (50 GB)
coldToFrozenDir Archive destination /archive/web_logs

Monitoring Index Health

# Check index sizes
| dbinspect index=main
| stats sum(sizeOnDiskMB) AS size_mb by index

# Event count per index
| tstats count where index=* by index
| sort -count

# Daily ingestion volume
index=_internal source=*license_usage.log type=Usage
| stats sum(b) AS bytes by idx
| eval gb = round(bytes/1024/1024/1024, 2)
| sort -gb

Knowledge Objects

Knowledge objects are the building blocks for customizing and extending Splunk.

flowchart TB
    subgraph KO["Knowledge Objects"]
        SavedSearch["Saved Searches<br>Reports & Alerts"]
        EventType["Event Types<br>Event classification"]
        Tag["Tags<br>Field value labels"]
        FieldExtraction["Field Extractions<br>Regex & transforms"]
        Lookup["Lookups<br>External data enrichment"]
        Macro["Macros<br>Reusable SPL"]
        DataModel["Data Models<br>Structured schema"]
    end
    style SavedSearch fill:#3b82f6,color:#fff
    style EventType fill:#22c55e,color:#fff
    style Tag fill:#f59e0b,color:#fff
    style FieldExtraction fill:#8b5cf6,color:#fff
    style Lookup fill:#ef4444,color:#fff

Event Types

Event types let you categorize events with meaningful names.

# eventtypes.conf
[web_error]
search = index=web_logs status>=400

[auth_failure]
search = index=security action=failed sourcetype=auth_log

# Search by event type
eventtype=web_error
| stats count by host

Tags

Tags attach labels to field values, making it easy to search across event types.

# tags.conf
[eventtype=web_error]
error = enabled
web = enabled

[eventtype=auth_failure]
error = enabled
authentication = enabled

# Search by tag
tag=error
| stats count by eventtype

Data Models

Data models define structured schemas for your data. They power the Pivot interface and search acceleration.

Concept Description
Data model A schema defining fields and constraints
Dataset A table within a data model
Pivot A GUI-based analysis tool
Acceleration Pre-computed summaries for faster searches

Splunk Apps

App Directory Structure

$SPLUNK_HOME/etc/apps/my_app/
├── default/
│   ├── app.conf          # App metadata
│   ├── data/
│   │   └── ui/
│   │       ├── views/    # Dashboards
│   │       └── nav/      # Navigation
│   ├── savedsearches.conf # Saved searches
│   ├── macros.conf        # Macros
│   └── transforms.conf    # Lookup definitions
├── local/                  # User customizations
├── lookups/               # Lookup files
├── metadata/
│   └── default.meta       # Permission settings
└── bin/                   # Scripts

app.conf

[install]
is_configured = true

[ui]
is_visible = true
label = My Security App

[launcher]
description = Custom security monitoring application
version = 1.0.0
author = admin

Navigation (default.xml)

<nav>
  <view name="overview" default="true" />
  <view name="errors" />
  <view name="performance" />
  <collection label="Reports">
    <view name="daily_report" />
    <view name="weekly_report" />
  </collection>
</nav>

Popular Splunkbase Apps

App Description
Splunk Enterprise Security (ES) SIEM platform
Splunk IT Service Intelligence (ITSI) IT monitoring
Splunk Add-on for AWS AWS data ingestion
Splunk Add-on for Microsoft 365 M365 data ingestion
DB Connect Database integration

Configuration File Precedence

Splunk configuration files are layered, and higher layers override lower ones.

flowchart TB
    System["system/local<br>Highest priority"]
    AppLocal["app/local<br>User customizations"]
    AppDefault["app/default<br>App defaults"]
    SystemDefault["system/default<br>Lowest priority"]
    System --> AppLocal --> AppDefault --> SystemDefault
    style System fill:#ef4444,color:#fff
    style AppLocal fill:#f59e0b,color:#fff
    style AppDefault fill:#22c55e,color:#fff
    style SystemDefault fill:#3b82f6,color:#fff
Priority Path Description
1 (highest) system/local/ System-wide customizations
2 app/local/ Per-app customizations
3 app/default/ App default settings
4 (lowest) system/default/ Splunk factory defaults

Important: Always make customizations in local/ directories. Never modify default/ files directly.

Debugging with btool

# List effective configuration
/opt/splunk/bin/splunk btool inputs list --debug

# Check a specific stanza
/opt/splunk/bin/splunk btool props list my_sourcetype --debug

Monitoring Splunk Health

Internal Log Queries

# Splunk internal errors
index=_internal log_level=ERROR
| stats count by component
| sort -count

# Search performance
index=_audit action=search
| stats avg(total_run_time) AS avg_runtime, max(total_run_time) AS max_runtime by user
| sort -avg_runtime

# License usage
index=_internal source=*license_usage.log type=Usage
| timechart span=1d sum(b) AS bytes
| eval gb = round(bytes/1024/1024/1024, 2)

Hands-On: Administrative Tasks

# 1. Index health check
| rest /services/data/indexes
| table title, currentDBSizeMB, maxTotalDataSizeMB, totalEventCount
| eval usage_pct = round(currentDBSizeMB/maxTotalDataSizeMB*100, 1)
| sort -currentDBSizeMB

# 2. User activity audit
index=_audit action=search
| stats count AS search_count, latest(_time) AS last_active by user
| eval last_active = strftime(last_active, "%Y-%m-%d %H:%M")
| sort -search_count

# 3. Scheduled search status
| rest /services/saved/searches
| where is_scheduled=1
| table title, cron_schedule, next_scheduled_time, dispatch.earliest_time, dispatch.latest_time
| sort next_scheduled_time

# 4. Forwarder connection status
index=_internal source=*metrics.log group=tcpin_connections
| stats latest(connectionType) AS type, latest(version) AS version, latest(fwdType) AS fwd_type by hostname
| sort hostname

Summary

Concept Description
Roles Define user permissions and access levels
Indexes Data storage with retention and size policies
Knowledge objects Event types, tags, macros, and more
Apps Packaged collections of dashboards, searches, and configs
Config precedence local/ overrides default/
btool CLI tool for debugging configuration

Key Takeaways

  1. Use roles to enforce proper access control
  2. Separate indexes by data type for better management
  3. Always save customizations in local/ directories
  4. Use btool to diagnose configuration issues

Exercises

Exercise 1: Basic

Create an analyst role that has access only to the main and web_logs indexes.

Exercise 2: Applied

Create event types and tags so that security-related events (authentication failures, permission errors) can be found with tag=security.

Exercise 3: Challenge

Build a custom Splunk app from scratch with a directory structure including app.conf, navigation, dashboards, lookups, macros, and saved searches. Verify that Splunk recognizes it.

References

Coming up next: In Day 10, you will tackle the final project -- building a Security Operations Center (SOC) dashboard that brings together everything you have learned over the past nine days.