SPLUNK INTERVIEW QUESTIONS AND ANSWERS 2023

SPLUNK INTERVIEW QUESTIONS AND ANSWERS

1)  What is Splunk?

Ans. Splunk is the tool being used across a broad variety of industries, it’s a data analytics and a SIEM tool, As a data analytics tool is used for Searching, Monitoring, and examining machine-generated data in a web interface for troubleshooting and for data visualization and as a SIEM it is used to collect the real-time logs from various devices such as security devices, Network devices, Servers, and Application and enables you for keeping the logs for longs time for forensics and compliance.

2) What are common port numbers used by Splunk?

Ans. Below are common port numbers used by Splunk,

however, you can change them if it required 

Service                                        Port number Used

Splunk Web Port:                              8000

Splunk Management Port:                  8089

Splunk Indexing Port:                         9997

Splunk Index Replication Port           8080

Splunk network port:                         514 (Used to get data in from network port i.e. UDP data)

KV store                                             8191

3) What are the components of Splunk and Splunk architecture?

Ans. TIP: Keep the answer short and straight.

1)  Search head  – provides GUI for searching
2) Indexer – indexes machine data
3) Forwarder -Forwards logs to Indexer
4) Deployment server -Manges Splunk components in a distributed environment

4) Which is the latest Splunk version in use?

Ans. Splunk Version 8.2

5) What is a Splunk indexer?

Ans. The indexer is the core component of Splunk architecture once it receives the logs from forwarders or end devices it parses the logs compress the logs and write the logs on the disk which is called indexing, Indexer can keep the logs for a long time for compliance, auditing and forensics perspective, Storage where Splunk stores the logs called and Bucket.

1- Indexing incoming data.
      2- Searching the indexed data.

6) What are the stages of Splunk indexing?

Ans.

7) What is a Splunk forwarder and What are the types of Splunk forwarders?

TIP: Keep the answer short and straight.

Ans. There are two types of Splunk forwarder:
a) Universal forwarder – Universal forwarder is a lightweight client that can be installed on the end servers like Windows, Linux, Mac, etc. which collects the logs locally and apply metadata fields like source, host, and source type and send it to Heavy Forwarders for further process or directly to Indexer, then the main advantage of installing Universal forwarders is that I can collect full logs on the host like memory utilization logs, registry logs, CPU logs and all. Universal forwarders do not parse the events it send the events to the heavy forwarders for parsing or to directly Indexer for further parsing and searching of the data.

b) Heavyweight forwarder(HWF) – a full instance of Splunk with advanced functionality. Generally works as a remote collector, intermediate forwarder, and possible data filter because they parse data, they are not recommended for production systems.

8) what are the most important configuration files used in Splunk?

Ans.

inputs.conf
transforms.conf
server.conf

props.conf
indexes.conf

9)   What are the types of Splunk licenses? 

Ans.  Enterprise license
 Free license
 Forwarder license
 Beta license
 Licenses for search heads (for distributed search)
 Licenses for cluster members (for index replication)

10)  What is the Splunk app?

Ans. Splunk app is a container/directory of configurations, dashboards, searches etc. in Splunk

11)  Where does Splunk’s default configuration is stored?

Ans. $splunkhome/etc/system/default

12) What features are not available in Splunk free? 

Ans. Splunk free lacks these features: 
authentication and scheduled searches/alerting
distributed search
forwarding in TCP/HTTP (to non-Splunk)
deployment management.

13) what happens if the license master is unreachable?

Ans. The license slave will take up a 24-hour timer, and the search will undoubtedly be blocked on the license slave (though indexing continues). users Won’t manage to search data in that slave until it can reach the license master again. 

14) what is a summary index in Splunk?

Ans. The Summary index could be the default summary index (the index that Splunk Enterprise uses if you do not indicate another one).
If you intend to run a variety of summary index reports you will need to generate additional summary indexes.

15) Wat is Splunk DB connect?

Ans. Splunk DB Connect is just a generic SQL database plugin for Splunk that enables you to easily integrate database information with Splunk queries and reports.

16) What is the difference between stats vs transaction command?

Ans.

The transaction command is best in two specific cases:

1. Unique id (from more than one field) alone isn’t sufficient to discriminate between two transactions. This is actually the case
2. once the identifier is reused, like, web sessions identified by cookie/client IP. In this case, time span or pauses
3. may also be used to segment the data into transactions. In other cases when an identifier is reused, say in DHCP logs,
4. a certain message may identify the first or end of a transaction.
5. When it is desirable to begin to see the raw text of the events combined rather than analysis on the constituent fields of the events.

In other cases, it’s usually better to make use of stats as the performance is higher, especially in a distributed search environment.
Often there is a distinctive id and stats may be used.

17) Can you write down a general regular expression for extracting IP addresses from logs?

Ans. You can find multiple ways we could extract IP addresses from logs. Listed here are a few examples.
Regular Expression for extracting ip address:

rex field=_raw  “(?<ip_address>\d+\.\d+\.\d+\.\d+)”   
                                  OR
rex field=_raw  “(?<ip_address>([0-9]{1,3}[\.]){3}[0-9]{1,3})”

18) How to troubleshoot Splunk performance issues?

Ans. The solution to the question will be very wide but basically, the interviewer will be trying to find these keywords in the interview:
-Check  splunkd.log for almost any errors
-Check server performance issues i.e. CPU/memory usage, disk i/o, etc.
-Install SOS (Splunk on Splunk) app and check for warnings and errors in the dashboard
-check the number of saved searches currently running and their system resources consumption
– install Firebug, which really is a Firefox extension. After it’s installed and enabled, log into Splunk (using Firefox), open firebug’s panels,
and switch to the net panel (you will have to enable it). The Net panel will reveal the HTTP requests and responses alongside
enough time spent on each. This provides you with a lot of information quickly over which requests are hanging Splunk for a few seconds,
and which are blameless. etc.

19) What are buckets? explain Splunk bucket lifecycle?

Ans. Splunk places indexed data in directories, called as “buckets”. It is physically a directory containing events of a certain period.

• Hot bucket: It keeps the latest data for fast searching in Splunk.
• 2. Warm bucket: Data moves from a hot bucket and comes into a warm bucket, data moved to this bucket has a slow searching than Hot bucket.
• 3.Cold bucket: Data moves from warm bucket to cold bucket for long time logs retention, it has slower searching than the hot and warm bucket.
• Frozen bucket: This bucket can be the external disk like hard disk, taps, SAN, and NAS, this is a kind of Archival.
• 5.Thawed bucket: In order to retrieve the data from the frozen bucket (Archival) data comes in Thawed bucket to make it searchable in Splunk.

20)   What is the different between stats and event-stats commands?

Ans. Stats command generates summary statistics of existing fields in your search results and saves them as values in new fields.
Eventstats is comparable to the stats command, except that aggregation email address details are added inline to each event and as long as the aggregation is pertinent compared to that event. event-stats computes the requested statistics like stats but aggregates them to the first raw data.

21)  Who are the biggest direct competitors to Splunk?

Ans. logstash, Loggly, Loglogic, sumo logic etc..

22) how does Splunk determine 1 day, from a licensing perspective?

Ans. Midnight to midnight on the clock of the license master.

23)  Splunk licenses specify what ?

Ans. How much data you can index per calendar day.

24)  What is a command for restarting just the Splunk daemon?

Ans. Splunk start Splunkd

 25)  What is a command for restarting just the Splunk web server?

Ans. Splunk start splunk-web 

26) What is Command to enable Splunk to boot start?

Ans. $SPLUNK_HOME/bin/splunk enable boot-start

27)  What is command to check for running splunk processes on unix/Linux ?

Ans. ps aux | grep splunk 

28) What is sourcetype in splunk?

Ans. Source type is Splunk way of identifying data

29)  How to disable Splunk boot start?

Ans. $SPLUNK_HOME/bin/splunk disable boot-start

30) What is source type in Splunk?

Ans. Source type is Splunk way of identifying data

31) How to reset Splunk admin password?

Ans. To reset your password log in to the server on which Splunk is installed and rename the password file at the below location and then restart Splunk. After the restart, you can log in using the default username: admin password:changeme

$splunk-home\etc\passwd 

32)  How to disable Splunk launch message?

Ans. Set value OFFENSIVE=Less in splunk_launch.conf

33) What is btool or how will you troubleshoot Splunk configuration files?

Ans. Splunk btool is really a command-line tool that helps us to  troubleshoot configuration file issues or just see what values are being
used by your Splunk Enterprise installation in the existing environment.

34)  How to clear Splunk search history?

Ans. Delete the following file on Splunk server
$splunk_home/var/log/splunk/searches.log

35) What is the difference between the Splunk app and Splunk add on?

Ans. Basically, both contains preconfigured configuration and reports etc. but Splunk addon do not need a visual app. Splunk apps
have preconfigured visual app  

36) What is .conf files precedence in Splunk?

Ans. File precedence is as follows:

1. System local directory — highest priority
2. App local directories
3. App default directories
4. System default directory — lowest priority

36) what is fishbucket or what is fishbucket index?

Ans. Its a directory or index at default location /opt/splunk/var/lib/splunk.It contains seek pointers and CRCs for the files you
are indexing, so splunkd can tell if it’s read them already. We can access it through GUI by looking for “index=_thefishbucket”

37 . How do i exclude some events from being indexed by Splunk?

Ans. This can be done by defining a regex to match the necessary event(s) and send everything else to the null queue. Here is a basic
example that will drop everything except events that contain the string login In props.conf:

——————————————————————–
<code>[source::/var/log/foo]

# Transforms must be applied in this order

# to make sure events are dropped on the

# floor prior to making their way to the

# index processor

TRANSFORMS-set= setnull,setparsing

</code>
————————————————————————-
In transforms.conf

————————————————————————————–
[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

38) How to set the default search time in Splunk 6?

Ans. To do this in Splunk Enterprise 6.0, use ui-prefs.conf. If you set the value in $SPLUNK_HOME/etc/system/local, all your users
should see it as the default setting. For example, if your $SPLUNK_HOME/etc/system/local/ui-prefs.conf file includes:

1.   [search]

2.   dispatch.earliest_time = @d

3.   dispatch.latest_time = now  

39) How can i tell when Splunk is finished indexing a log file?

By watching  data from splunk’s metrics log in real-time.

index=”_internal” source=”*metrics.log” group=”per_sourcetype_thruput” series=”&lt;your_sourcetype_here&gt;” |
eval MB=kb/1024 | chart sum(MB)

or to watch everything happening split by sourcetype….

index=”_internal” source=”*metrics.log” group=”per_sourcetype_thruput” | eval MB=kb/1024 | chart sum(MB) avg(eps) over series

40. What is difference between search head pooling and search head clusttering?

Both are features provided Splunk for high availability of Splunk search head in case anyone search head goes down. Search 
head cluster is newly introduced and search head pooling will be removed in next upcoming versions. Search head cluster is
managed by the captain and the captain controls its slaves. Search head cluster is more reliable and efficient than search head pooling.

41. What is dispatch directory?

$SPLUNK_HOME/var/run/splunk/dispatch has a directory for each search that’s running or has completed. For instance,
a directory named 1434308943.358 will contain a CSV file of its search results, a search.log with information regarding the search
execution, and other stuff. Utilizing the defaults (which you can override in limits.conf), these directories is going to be deleted 10 minutes
following the search completion – unless an individual saves the search results, by which case the outcomes is going to be deleted after 7 days.

42. How would you handle/troubleshoot Splunk license violation warning error?

License violation warning means Splunk has indexed more data than our purchased license quota. We have to identify which
index/source type has received more data recently than the usual daily data volume. We can check into Splunk license master pool available quota and identify the pool for which violation is occurring. Once we all know the pool for which we are receiving more data then we have to identify the utmost effective source type for which we are receiving more data than usual data. Once the foundation type is identified
then we have to locate outsource machine that is sending a huge number of logs and the root reason for exactly the same and troubleshoot
accordingly.

43. If I want add/onboard folder access logs from a windows machine to Splunk how can I add same?

Here are steps to include folder access logs to Splunk
1.Enable Object Access Audit through group policy on a windows machine on which folder is found
2. Enable auditing on the precise folder which is why you wish to monitor logs
3.Install Splunk universal forwarder on a Windows machine
4.Configure universal forwarder to send security logs to Splunk indexer

44. For what purpose inputlookup and outputlook are used in Splunk search?

inputlookup command returns the whole lookup table as search results.
For example
…| inputlookup lookuptabllename returns a search result for every row in the table lookup which has two field
​values:
• host
• machine_type.

Outputlookup  outputs the current search results to a lookup table on the disk.
For example
…| outputlookup lookup.csv saves all the results into lookup.csv.

45. What is the difference between Splunk SDK and Splunk framework?

Splunk SDKs are made to enable you to develop applications from the floor up and not require Splunk Web or any
components from the Splunk App Framework. They’re separately licensed for your requirements from the Splunk Software and do not alter
the Splunk Software. Splunk App Framework resides within Splunk’s web server and permits you to customize the Splunk Web
UI that comes with the merchandise and develops Splunk apps utilizing the Splunk web server. It is an essential area of the features
and functionalities of Splunk Software, which doesn’t license users to modify anything in the Splunk Software.

46. How Splunk avoids duplicate indexing of logs?

At indexer splunk keeps track of indexed events in a directory called fish buckets (default location /opt/splunk/var/lib/splunk).
It contains seek pointers and CRCs for the files you are indexing, so splunkd can tell if it has read them already. – See more at:
http://www.learnsplunk.com/splunk-indexer-configuration.html#sthash.t1ixi19P.dpuf.

47. What is mapreduce algorithm?

Maprduce algorithm is the key behind Splunk fast data searching speed.It’s an algorithm typically employed for batch-based large

  scale parallelization. It’s inspired by functional programming’s map() and reduce () functions.

Career Scope in SIEM | ArcSight | Splunk