Today I invite you to do a hands-on exercise and check out how PGO works in Golang.

This demo is based on this repo, which I encourage you to clone: fira.curie/go-pgo-app.

Prerequisites

You would need the following tools installed on your machine:

  • Docker or Podman - if you’re using podman, you need to enable insecure registry (see below).
  • kind to provision a local k8s cluster in a container.
  • Helm - for deploying our app, Pyroscope, and optionally Grafana.
  • ohayou - for simulating production load.
  • profilecli - for collecting profile data from the app.

Assumptions

I assume that you know how to work with git, bash, kubernetes, helm, and docker. If you don’t, you can check out the official documentation and helm documentation.

I also assume that you have a basic understanding of how Go works and what is PGO. If not - see further reading section.

The purpose of this workshop is to give you a hands-on experience with PGO in Go and create a playground to experiment with different types of loads.

If you have any questions or encounter any problems, feel free to email me!

Let’s go!

Setup cluster

Create kind(kubernetes in docker) cluster and create a registry container for storing and deploying local images.

1

./kind-cluster-with-registry.sh

Deploy Pyroscope (optionally with Grafana)

Add helm chart

1
2

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Deploy pyroscope

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# create namespace
kubectl create namespace pyroscoping
# deploy pyroscope
helm -n pyroscoping install pyroscope grafana/pyroscope

# optionally with grafana
helm upgrade -n pyroscoping --install grafana grafana/grafana \
  --set image.repository=grafana/grafana \
  --set image.tag=main \
  --set env.GF_FEATURE_TOGGLES_ENABLE=flameGraph \
  --set env.GF_AUTH_ANONYMOUS_ENABLED=true \
  --set env.GF_AUTH_ANONYMOUS_ORG_ROLE=Admin \
  --set env.GF_DIAGNOSTICS_PROFILING_ENABLED=true \
  --set env.GF_DIAGNOSTICS_PROFILING_ADDR=0.0.0.0 \
  --set env.GF_DIAGNOSTICS_PROFILING_PORT=6060 \
  --set-string 'podAnnotations.pyroscope\.grafana\.com/scrape=true' \
  --set-string 'podAnnotations.pyroscope\.grafana\.com/port=6060'

If you’re using podman instead of docker

You need to explicitly enable insecure registry

1
2
3
4
5
6

cat > /etc/containers/registries.conf.d/local.conf <<EOF
[[registry]]
location = "localhost:5001"
insecure = true

EOF

First iteration of the Go app

For the purposes of this demonstration, we are going to use /render http handler, but you can add your own kind of workload here.

First of all we need to build our app.

  • You can notice that we have set an argument for PGO_FILE in the Dockerfile.
  • By default go looks for default.pgo file, but it also accepts off as a parameter to disable PGO.
  • BUILD_VERSION is used as a tag for the image in order to track the changes of the certain version in Pyroscope. You could do this at runtime as well.
1

docker build --build-arg PGO_FILE=off --build-arg BUILD_VERSION=1.0.0 -t localhost:5001/go-pgo-app:1.0.0 .

Push the image to the local registry we deployed previously in a docker container and deploy the app with helm.

1
2
3

docker push localhost:5001/go-pgo-app:1.0.0

helm install go-pgo-app ./chart/

Simulate production load

We need some workload on our server. Let’s send some .md files its way.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# port-forward to the service (internal port is 80)
kubectl port-forward svc/go-pgo-app-chart 8080:80

# use ohayout to send a lot of requests and create production-like load
# -q parameter is for query rate (qps)
# -n is for total number of requests
oha -n 2000 -q 500 -m POST -D $(pwd)/awesome.md "http://localhost:8080/render"

# also available
oha -n 2000 -q 500 -m POST "http://localhost:8080/pi?num=10000" # num is number of iterations
oha -n 2000 -q 500 -m POST "http://localhost:8080/fibonacci?num=100000" # num is number of iterations

Don’t forget to save this data for comparison.

Collect profile data

Check pyroscope UI where you can see the traces of the requests.

1
2

# open localhost:4040 in your browser to view traces
kubectl --namespace pyroscoping port-forward svc/pyroscope 4040:4040

Collect the profile data from the app for the optimized build into a file named go-pgo-app-v1.0.0.pgo.

You could also notice that we are passing the same version in iteration parameter as we did in the BUILD_VERSION.

1

profilecli query merge --query='{service_name="simple.golang.app", iteration="1.0.0"}' --profile-type="process_cpu:cpu:nanoseconds:cpu:nanoseconds" --from="now-1d" --to="now" --output=pprof=./go-pgo-app-v1.0.0.pgo

This will create a file named go-pgo-app-v1.0.0.pgo in your current directory containing trace data for the first iteration of the app.

Second optimized iteration of the Go app

Now let’s rebuild the app with the collected profile data. As you can see we are passing the PGO_FILE as an argument to create an optimized build. Also we should increment BUILD_VERSION and image tag to track the new app traces.

1
2
3
4
5

# build the image with the profile data
docker build --build-arg PGO_FILE=./go-pgo-app-v1.0.0.pgo --build-arg BUILD_VERSION=1.0.1 -t localhost:5001/go-pgo-app:1.0.1 .

# push the image version
docker push localhost:5001/go-pgo-app:1.0.1

Now we can either upgrade the helm chart with the new image or deploy the new version alongside. Set the image tag to the new version.

1
2
3

helm upgrade --set image.tag=1.0.1 go-pgo-app ./chart/
# or create new one
helm install --set image.tag=1.0.1 go-pgo-app-v1-0-1 ./chart/

Check the results

1
2
3

# and port forward to 8081
kubectl port-forward svc/go-pgo-app-v1-0-1-chart 8081:80
oha -n 2000 -q 500 -m POST -D $(pwd)/awesome.md "http://localhost:8081/render"

Now compare the two runs.

Cleanup

1
2

kind delete cluster
docker rm -f kind-registry

Further reading