DEV Community

Pascal Matthiesen
Pascal Matthiesen

Posted on • Edited on

Cloudflare Logs: Not Just for Breakfast Anymore 🥞

#kubernetes #infrastructureascode

Alright, log lovers, let's get this observability party started 🎉. We're about to turn your Cloudflare logs from wallflowers into the life of the data analysis party 🥳.

Challenge 😕: Cloudflare Logpush delivers valuable data, but it might not be in the most usable format for further analysis.

Solution 💡: Enter Vector and Loki, the dream team 🏆. We'll leverage Vector's built-in HTTP server to accept your Cloudflare log lines, process them, and then ship them off to Loki for storage and analysis 🚀.

Why Bother? 🤔

Find and fix issues faster: Your logs will be organized and searchable, making troubleshooting a breeze.

Optimize performance: Identify bottlenecks and fine-tune your applications like a pro.

Boost security: Detect threats and protect your digital assets.

Get Ready to Geek Out!

We'll dive into the technical details soon, so get your command line fingers ready. You'll learn how to:

  • Set up Vector to transform and forward your logs.
  • Get Loki ready to ingest and analyze your data.
  • Configure Cloudflare Logpush to send logs via HTTP.

These commands give your Kubernetes cluster a VIP backstage pass 🎫 to the hottest logging tool in town: Vector! ✨

  • helm repo add vector https://helm.vector.dev: This tells Helm (your trusty Kubernetes package manager) where to find Vector's exclusive swag, like adding a secret app store to your phone 🤫.
  • helm repo update: This command refreshes Helm's knowledge, making sure you have access to the freshest Vector releases. Think of it as checking for those "new app" notifications on your phone 📲. 
helm repo add vector https://helm.vector.dev
helm repo update
Enter fullscreen mode Exit fullscreen mode

This Kubernetes Secret ("vector") is the Fort Knox 🔒 of your Vector logging tool, guarding a super-secret HTTP password ("123abc") 🤫. But hold on, Captain Obvious here 🦸 – change that password before some sneaky cyber ninja 🥷 steals your precious logs!

apiVersion: v1
kind: Secret
metadata:
  name: vector
  namespace: vector
stringData:
  HTTP_PASSWORD: "123abc"
Enter fullscreen mode Exit fullscreen mode

This Vector configuration sets up a secure gateway 🚪 for your Cloudflare logs. It grabs the logs, translates them into a format Loki understands 🗣️, and then sends them off for safekeeping 📦. It's like a trusty butler for your website data 🤵.

envFrom:
  - secretRef:
      name: vector
service:
  ports:
    - name: http
      port: 3000
  enabled: true
role: Stateless-Aggregator
customConfig:
  acknowledgements:
    enabled: true
  sources:
    in:
      type: "http"
      address: 0.0.0.0:3000
      strict_path: false
      encoding: text
      path: /ingest
      auth:
        username: cloudflare
        password: "${HTTP_PASSWORD}"
  transforms:
    parse_json:
      type: remap
      inputs:
        - "in"
      source: |
        . = parse_json!(.message)
        .timestamp = from_unix_timestamp!(.EventTimestampMs, unit: "milliseconds")
  sinks:
    out:
      type: "loki"
      tenant_id: "0:0"
      encoding:
        codec: "json"
      remove_timestamp: false
      out_of_order_action: accept
      labels:
        job: cloudflare
      endpoint: http://loki-gateway.monitoring.svc.cluster.local
      inputs:
        - "parse_json"
Enter fullscreen mode Exit fullscreen mode

This command summons the Helm installer 🧙‍♂️ to conjure up Vector on your Kubernetes cluster:

helm install vector vector/vector \
  --namespace vector \
  --create-namespace \
  --values values.yaml
Enter fullscreen mode Exit fullscreen mode

It's like giving Helm a magic spell book 🪄, pointing to the Vector spell page, and telling it where to build Vector's cozy little home 🏡 (in the "vector" namespace) with the customizations you specified in the "values.yaml" file. Now you're ready to wrangle those logs! 🧹🪵

Now, let's teach Cloudflare to share those juicy logs with Vector! 🤝 Just replace the placeholders and run this command in your terminal:

curl -X POST "https://api.cloudflare.com/client/v4/accounts/<YOUR_CF_ACCOUNT_ID>/logpush/jobs" \
  -H "Authorization: Bearer <YOUR_CF_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
      "name": "vector-logs",
      "destination_conf": "<YOUR_VECTOR_ENDPOINT>?header_Authorization=Basic%20<BASE64_ENCODED_AUTH>",
      "dataset": "workers_trace_events",
      "enabled": true
  }'
Enter fullscreen mode Exit fullscreen mode

Replace these placeholders:

  • : Your Cloudflare account ID 🔢
  • : Your Cloudflare API token 🗝️
  • : The full URL to your Vector's HTTP endpoint (e.g., https://vector.yourdomain.com/ingest) 🌐
  • : Your Vector username and password encoded in Base64 format 🔒 (echo -n "$username:$password" | base64)

That's it! Cloudflare will start pushing logs to Vector, and Vector will whisk them off to Loki for safekeeping and analysis. 💨 Now you're a log analysis wizard! 🧙‍♂️

And with that, the logs have been tamed... for now. 😈 But rest assured, they'll be back with a vengeance, just like my insatiable craving for pizza. 🍕 If you enjoyed this wild ride through log analysis, join me next time as I tackle more tech challenges, share hilarious startup fails, and maybe even reveal my secret pizza recipe (just kidding...or am I? 😉).

Top comments (0)