Prometheus and localhost

#ops #prometheus #go

Prometheus scrapping needs to be able to access the /metrics handler you want to instrument.

So, I have a Prometheus server running behind a NAT (aka at home) and I want metrics from my public renter server. This process expose some metrics on http://localhost:2019/metrics.

Sure, I can push them, but then I have 99 other problems

Sure, I could expose them on http://0.0.0.0:2019 but then it open bar and I don’t like it.

Sure, I could add an IP rule but I can’t grantee my NATed public IP won’t change.

Sure, I could setup a double reverse gateway but come on, why is this so complicated.

My thinking was:

  • metrics are exposed on server’s localhost
  • I have ssh access to this host
  • The prometheus scrapping only want a single handler

So…

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"os/exec"
	"time"
)

type Server struct {
	TargetHost string
}

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {

	cmd := exec.Command("ssh", s.TargetHost, "curl", "http://127.0.0.1:2019/metrics")
	out, err := cmd.Output()
	if err != nil {
		fmt.Printf("error: %v\n", err)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}

	w.WriteHeader(http.StatusOK)
	w.Write(out)
}

func main() {
	bindAddr := os.Getenv("PROMSSHP_BIND")
	if bindAddr == "" {
		bindAddr = ":8090"
	}

	target := os.Getenv("PROMSSHP_TARGET") // i.e "myuser@myserver"

	s := &http.Server{
		Addr: bindAddr,
		Handler: &Server{
			TargetHost: target,
		},
		ReadTimeout:  60 * time.Second,
		WriteTimeout: 60 * time.Second,
	}
	log.Fatal(s.ListenAndServe())
}

Problem solved, I guess. Prometheus can scrap on its own localhost and I don’t have to maintain anything.

Start it with systemd, then forget about it.

[Unit]
Description=Prometheus metrics SSH proxy
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal

; User and group the process will run as.
User=myuser
Group=myuser

ExecStart=/home/proullon/go/bin/promsshp

[Install]
WantedBy=multi-user.target
← prev next →