Prometheus and localhost
Prometheus scrapping needs to be able to access the /metrics
handler you want to instrument.
So, I have a Prometheus server running behind a NAT (aka at home) and I want metrics from my public renter server. This process expose some metrics on http://localhost:2019/metrics
.
Sure, I can push them, but then I have 99 other problems
Sure, I could expose them on http://0.0.0.0:2019
but then it open bar and I don’t like it.
Sure, I could add an IP rule but I can’t grantee my NATed public IP won’t change.
Sure, I could setup a double reverse gateway but come on, why is this so complicated.
My thinking was:
- metrics are exposed on server’s localhost
- I have ssh access to this host
- The prometheus scrapping only want a single handler
So…
package main
import (
"fmt"
"log"
"net/http"
"os"
"os/exec"
"time"
)
type Server struct {
TargetHost string
}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
cmd := exec.Command("ssh", s.TargetHost, "curl", "http://127.0.0.1:2019/metrics")
out, err := cmd.Output()
if err != nil {
fmt.Printf("error: %v\n", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
w.Write(out)
}
func main() {
bindAddr := os.Getenv("PROMSSHP_BIND")
if bindAddr == "" {
bindAddr = ":8090"
}
target := os.Getenv("PROMSSHP_TARGET") // i.e "myuser@myserver"
s := &http.Server{
Addr: bindAddr,
Handler: &Server{
TargetHost: target,
},
ReadTimeout: 60 * time.Second,
WriteTimeout: 60 * time.Second,
}
log.Fatal(s.ListenAndServe())
}
Problem solved, I guess. Prometheus can scrap on its own localhost
and I don’t have to maintain anything.
Start it with systemd, then forget about it.
[Unit]
Description=Prometheus metrics SSH proxy
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
[Service]
Restart=on-abnormal
; User and group the process will run as.
User=myuser
Group=myuser
ExecStart=/home/proullon/go/bin/promsshp
[Install]
WantedBy=multi-user.target