Jagadish Rajasekar

Posted on Sep 6, 2025

🔹 A. Basics & Architecture (Q1–Q10)

Q1. What is Splunk?

👉 A platform for searching, monitoring, and analyzing machine data (logs).

Q2. What are the main components of Splunk?

Forwarder → collects & sends logs.
Indexer → stores and indexes data.
Search Head → query interface.

Q3. What is Splunk Enterprise vs Splunk Cloud?

Enterprise → self-hosted.
Cloud → SaaS offering.

Q4. What is a Universal Forwarder?

👉 Lightweight agent installed on servers to send logs to Splunk indexers.

Q5. What is an Index in Splunk?

👉 Logical repository where data is stored after parsing.

Q6. What is a Sourcetype?

👉 Identifies the format of incoming data (Apache logs, JSON, etc.).

Q7. How does Splunk store data?

👉 Stores events in indexes, split into raw data + index metadata.

Q8. What is Search Head Clustering?

👉 Multiple search heads for high availability and load balancing.

Q9. What is Indexer Clustering?

👉 Replication of indexed data for fault tolerance.

Q10. What is Deployment Server?

👉 Manages configuration deployment to forwarders.

🔹 B. SPL (Search Processing Language) Basics (Q11–Q20)

Q11. How do you search logs in Splunk?

index=app_logs error

Q12. How do you filter by time range?

👉 Use earliest and latest:

spl index=web earliest=-1h latest=now

Q13. How do you count events?

spl index=app_logs | stats count

Q14. How do you find top values of a field?

spl index=app_logs | top status_code

Q15. How do you calculate average response time?

spl index=perf_logs | stats avg(response_time)

Q16. How do you group by endpoint?

spl index=perf_logs | stats avg(response_time) by endpoint

Q17. How do you find 95th percentile response time?

spl index=perf_logs | stats perc95(response_time)

Q18. How do you filter out null values?

spl index=app_logs status_code!=null

Q19. How do you join two searches?

spl index=orders | join order_id [ search index=payments ]

Q20. How do you search for errors in the last 24 hours?

spl index=app_logs error earliest=-24h

🔹 C. Dashboards & Visualization (Q21–Q30)

Q21. What is a Splunk Dashboard?

👉 Visual representation of search results.

Q22. How do you create an alert from a search?

👉 Save search → configure alert conditions.

Q23. What is a Panel in dashboards?

👉 A single chart or visualization inside a dashboard.

Q24. What visualizations does Splunk support?

👉 Time charts, pie, bar, scatter, heatmaps.

Q25. How do you schedule a search?

👉 Save search → set schedule (every 5 mins, hourly, etc.).

Q26. What is a Report in Splunk?

👉 A saved search that can be scheduled and shared.

Q27. How do you create a drilldown in dashboards?

👉 Use tokens to pass parameters between panels.

Q28. What is a Real-time Dashboard?

👉 Continuously updates with incoming events.

Q29. How do you monitor system KPIs with Splunk?

👉 Create dashboards for CPU, memory, TPS, error % from logs.

Q30. How do you share dashboards?

👉 Via app context or export as PDF.

🔹 D. Advanced SPL Queries (Q31–Q40)

Q31. How do you extract fields dynamically?

👉 Use rex:

spl index=app_logs | rex "user=(?<username>[^ ]+)"

Q32. How do you calculate throughput (TPS)?

spl index=perf_logs | timechart count span=1s

Q33. How do you find error rate?

spl index=app_logs | stats count(eval(status>=500)) as errors, count as total | eval error_rate=errors/total*100

Q34. How do you find slowest transactions?

spl index=perf_logs | sort - response_time | head 10

Q35. How do you detect anomalies?

spl index=perf_logs | timechart avg(response_time) | anomalydetection

Q36. How do you find logs with specific JSON field?

spl index=json_logs | spath path=user.id

Q37. How do you correlate two fields?

spl index=app_logs | stats count by status, endpoint

Q38. How do you create a pivot table?

👉 Use Splunk Pivot interface on data models.

Q39. How do you find peak load periods?

spl index=perf_logs | timechart span=1m count

Q40. How do you calculate 90th percentile latency per service?

spl index=perf_logs | stats perc90(response_time) by service

🔹 E. Troubleshooting & Real Scenarios (Q41–Q50)

Q41. How do you troubleshoot latency spikes?

👉 Check 95th percentile → filter by endpoint → correlate with errors.

Q42. How do you detect memory leaks with Splunk logs?

👉 Search for OutOfMemoryError and correlate with heap usage metrics.

Q43. How do you detect DB slowness?

👉 Search DB logs → filter queries with execution_time > threshold.

Q44. How do you detect API failures during load test?

👉 Search by status_code >= 500 → group by endpoint.

Q45. How do you detect thread deadlocks?

👉 Search for Found one Java-level deadlock in JVM logs.

Q46. How do you create correlation between JMeter and Splunk logs?

👉 Use transaction IDs to join JMeter logs with app logs.

Q47. How do you alert on error bursts?

👉 Monitor error count in rolling 5-minute windows.

Q48. How do you optimize Splunk search performance?

Narrow time window.
Use indexed fields.
Avoid wildcards at start of searches.

Q49. How do you explain Splunk to a CIO vs Developer?

CIO → business dashboards, uptime, compliance.
Developer → detailed logs, debugging traces.

Q50. What are Splunk’s limitations?

👉 Expensive at scale, heavy log ingestion costs, less AI automation compared to Dynatrace.

✅ Final Takeaway

For Splunk interviews, focus on:

Architecture (forwarder, indexer, search head)
SPL queries (stats, timechart, rex, spath)
Dashboards & alerts
Troubleshooting scenarios (latency spikes, OOM errors, DB slowness)
Optimization & cost considerations

👉 Always connect answers to real-world performance engineering use cases: e.g., “I used Splunk SPL to correlate JMeter test failures with backend exceptions.”