invalid upgrade response error when sending requests to the K8s API server

[cerulean storm]

I have a Kubernetes cluster running on Rancher Desktop. I want to use Ballerina to send requests to the Kubernetes API server. I am following https://kubernetes.io/docs/tasks/extend-kubernetes/http-proxy-access-api/

The steps I followed are as follows

1. Executed kubectl proxy --port=8080
2. Started the following Ballerina service and sent requests to it

import ballerina/http;

service / on new http:Listener(9090) {
    resource function get pod/[string pod]() returns json|error {
        http:Client k8sclient = check new ("http://localhost:8000");
        string path = "/api/v1/namespaces/default/pods/" + pod;
        json response = check k8sclient->get(path);
        return response;
    }
}

Everytime I start the Ballerina service, the first request to it fails with the following error.

In the client

invalid upgrade response: status code 200

In the Ballerina service

Running executable

error: Internal Server Error {"statusCode":500,"headers":{"Content-Length":["42"],"Content-Type":["text/plain; charset=utf-8"],"Date":["Wed, 24 May 2023 09:38:04 GMT"],"X-Content-Type-Options":["nosniff"]},"body":"invalid upgrade response: status code 200
"}
        at ballerina.http.2:createResponseError(http_client_endpoint.bal:707)
           ballerina.http.2:processResponse(http_client_endpoint.bal:640)
           ballerina.http.2.Client:processGet(http_client_endpoint.bal:286)

In the Kubectl proxy terminal window

E0524 15:08:04.795831   11381 upgradeaware.go:365] Proxy upgrade error: invalid upgrade response: status code 200
E0524 15:08:04.795865   11381 proxy_server.go:147] Error while proxying request: invalid upgrade response: status code 200

But the 2nd request onwards works without any issue.
Does anyone know why this happens and what can I do to avoid it?

[bold pewter]

Hi @cerulean storm,
Does this request work if you invoke it manually? eg - curl?

[cerulean storm]

When I use curl and send the request to the K8s API server, there are no errors and I get the response for all requests.
With Ballerina, the 2nd request onwards works. It's always the 1st request that fails (Updated the original post with this info)

[cerulean storm]

Found the issue. This was caused by the HTTP version. Ballerina's HTTP client uses HTTP/2 by default while curl uses HTTP/1.1 by default . That's why it worked with curl.

After I changed the Ballerina client HTTP version to 1.1, first request also works.

http:Client k8sclient = check new ("http://localhost:8000", { httpVersion: "1.1" });