Serving detection and segmentation models over gRPC

The web backend should not import PyTorch. We keep inference in its own gRPC service with the GPU, and the app talks to it over a thin contract. That way the model service can be restarted, scaled or swapped without touching the API.

The contract

service MLService {
  rpc Infer (InferRequest) returns (InferReply);
}
message InferRequest { bytes image = 1; float conf = 2; }
message Detection { int32 cls = 1; float score = 2;
                    repeated float bbox = 3; bytes mask_png = 4; }
message InferReply { repeated Detection dets = 1; }

Send masks as PNG bytes rather than raw arrays. It is a fraction of the size on the wire and every language decodes it for free.

Load the model once

class MLServicer(ml_pb2_grpc.MLServiceServicer):
    def __init__(self):
        self.model = build_model().eval().cuda()   # load ONCE

    @torch.inference_mode()
    def Infer(self, request, context):
        img = decode(request.image)
        out = self.model(img)
        return pack(out, conf=request.conf)

Things that bite

Bump grpc.max_receive_message_length; default 4MB drops full-res frames. Keep the servicer thread pool small and let the GPU serialise work, otherwise concurrent requests just thrash VRAM. And run a real warmup inference at startup, because the first CUDA call takes seconds and you do not want that latency landing on a user's first request.