fix: reduce memory usage of SBOM browser

[dervoeti]

The SBOM browser is OOMKilled regularly. Almost all /sbom/ traffic comes from crawlers, which request dozens of download links at once, every one of them starts its own cosign verify-attestation process.

Fixes added to reduce memory usage:

• Limit concurrent cosign runs to 4: A request waits up to 10s for a free slot, then gets a 503 with Retry-After: 30. 4 slots at about 5s per run still handle around 2900 downloads per hour, the worst hour so far had 645. And the SBOM browser should only be used for manual exploration. Automated SBOM checks in prod clusters for example would run via cosign directly, so they are not affected by this limit.
• Keep the SBOM as raw JSON, it is served as JSON download anyway, no need for pretty-printing
• Add a robots.txt that disallows /sbom/: The links already have rel='nofollow', but that's only a hint. This stops at least the crawlers that respect robots.txt.