adam's notes

Splunk Search Optimization

Search Optimization

Search Modes determine how much field data is returned, which affects how fast a search is returned.

  • 3 modes:
    • Smart Mode
      • default mode
      • balances speed and completeness
      • for transforming searches, behave like fast mode
      • for non-transforming searches, behaves like verbose mode
    • Fast Mode
      • prioritizes speed over completeness
      • field discovery is disabled
      • only returns info on default fields and fields required to fulfill search
    • Verbose Mode
      • prioritizes completeness over speed
      • returns all extracted fields, an event list, and an event timeline
      • slowest
  • Using time to filter events is the most efficient
    • followed by:
      1. index
      2. source
      3. host
      4. sourcetype

Splunk Search Scheduler

Search scheduler defines and manages when scheduled reports and alerts will run.

  • limit to the number of searches Splunk can run concurrently
    • determined by number of CPUs
  • when there are too many searches, the a priority is used to determine which search to run
    1. Ad hoc historical searches
      • e.g., searches run manually
    2. Manually scheduled searches with real-time scheduling
      • e.g., user-saved scheduled reports and alerts that use real-time scheduling
      • real-time is a scheduling mechanism (does not mean constantly running)
        • these have potentially to be skipped
    3. Manually scheduled searches with continuous scheduling
      • e.g., user-saved scheduled reports or alerts that use continuous scheduling
      • searches can be paused for extensive periods of time
    4. Automatically scheduled searches
      • e.g., auto-summarization reports

Search Acceleration Overview

  • time it takes to return results is affected by the number of events returned in a search
  • Splunk allows creation of summaries of event data
    • these are smaller segments of event data populated by background searches that only include the data needed to fulfill the search
    • when you run a search against a summary, it should complete much faster
  • 3 data summary creation methods:
    • Report acceleration
      • uses automatically created summaries to speed up completion time for reports
    • Summary indexing
      • uses manually enabled summaries
      • these indexes live separately from other indexes in deployment
    • Data model acceleration
      • accelerates all fields in a data model
      • easiest and most efficient option

Report Acceleration

  • used to accelerate individual reports
  • when an accelerated report is created, Splunk runs the report and populates and acceleration summary
  • acceleration summary is a stored search result populated every 10 minutes
  • summaries are stored as file chunks by time alongside buckets in your indexes
  • when the report is run, the events stored in the summary are used to return results
  • before accelerating a report:
    • search mode must be Smart or Fast
    • users must have the schedule search privilege
    • deleting all reports used by a summary, then the summary also gets deleted
    • must be aware of commands the report is using
      • 3 types:
        • Streaming commands
          • operate on events as they are returned
        • Non-streaming commands
          • wait until all events are returned by indexers before executing
        • Transforming commands
          • order results into a data table
      • accelerated reports must include a transforming command
      • any commands before a transforming command can only be streaming commands
      • any commands after a transforming command need to be non-streaming

Data Model Acceleration

  • creates a High-Performance Analytics Store which are files used to speed up searches
  • 2 types of data model acceleration
    • ad-hoc
      • takes place on data models that have not been accelerated
      • files created for an acceleration are stored on a search head
        • only exist when the user is actively using the pivot tool
      • first time opening a pivot runs on all the data in the data model
        • so may take long
      • pivot editor uses the summary data,
        • but reports created in the pivot editor will not
      • running ad-hoc acceleration is taxing on a deployment
      • only apply to one user
    • persistent
      • address the problem of ad-hoc acceleration
      • the file created for a persistent acceleration are stored alongside buckets on the indexers
        • exist as long as the data model exists
        • can be scoped to time ranges
        • maintained on an on-going basis
      • can be used by all users
  • restrictions
    • requires a user with accelerate data models privilege or admin role
    • private data models cannot be accelerated
    • once a data model is accelerated, it cannot be edited
    • only event data set hierarchies and search data set hierarchies that only use streaming commands can be used
  • how to:
    • Settings > Data Models > Edit > Accelerate > Summary Range
      • yellow lightning bolt means data model is accelerated
  • the files created for data model acceleration are called time-series index files (tsidx)
    • 2 main components
      • lexicon
        • is an alphanumerically ordered list of terms found in the data at index time
        • any fields extracted at index time are shown as key-value pairs in lexicon
      • posting list
        • is an array of pointers that match each term to events in the raw data files
    • similar to a complex index for a book
      • every word included in the book is found in the index

Datamodel Command

datamodel command displays the structure of data models and searches against them.

  • without arguments, generates a list of data models you have access to
    • in json
  • select a specific data model by specifying the data model id as argument
    • modelName
  • can also search with data set ID to filter further
  • can pull data through the data model in 2 ways:
    • use search option
      • returns a list of events and a field sidebar
    • use flat option
      • field sidebar does not indicate which data set defines each field
    • don’t have to specify the data set nam
      • automatically chooses a data set to use
      • defaults to first data set
  • | dataset [<data-model-id>] [<data-set-id>] [search|flat]
  • IDs are case sensitive
  • summariesonly=t option uses only data in data summaries
    • otherwise uses raw data

Tstats Command

tstats command generates statistics from tsidx files.

  • generating command
  • requires a stats function
  • | tstats <stats-func> [from datamodel=<data-model-name>] [where <searchQuery>] [by <field-list>]
  • summariesonly=t uses only data from data summaries
  • much faster that regular stats
    • but can only use fields found in tsidx files
  • can group by _time
    • behaves like the timechart command
    • span to set time buckets

Example

| tstats sum(apac.price) as Sales from datamodel=vsales.apac by apac.Vendor, apac.VendorCountry, apac.VendorCity

| search Sales > 200

Graph View

Backlinks