ContainersNVIDIADeepSeek-R1NVIDIALlama-3.1-Nemotron-70B-InstructNVIDIAPyTorch
CollectionsNVIDIAOmniverse Kit (FB)NVIDIADeepStream SDKNVIDIAOmniverse Kit App Streaming
ModelsNVIDIAStyleGAN3 pretrained modelsNVIDIAPeopleNetNVIDIATrafficCamNet
ResourcesNVIDIARiva Skills Quick Start
Helm ChartsNVIDIAGPU OperatorNVIDIANVIDIA NIM Operator
NVIDIA
NVIDIA
TensorRT Inference Server
Container
NVIDIA
NVIDIA
TensorRT Inference Server

TensorRT Inference Server provides a data center inference solution optimized for NVIDIA GPUs. It maximizes inference utilization and performance on GPUs via an HTTP or gRPC endpoint, allowing remote clients to request inference for any model that is being managed by the server, as well as providing real-time metrics on latency and requests.

LayerLabelCreated
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4ENV
LD_LIBRARY_PATH=/workspace/install/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
12/19/2019 8:08 PM UTC
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4ENV
PATH=//workspace/install/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
12/19/2019 8:08 PM UTC
sha256:607f3db169d0ccda7709b54faf01b748a175eeaecdafa7274123ecf77b96774dRUN
python -m pip install --user --upgrade pip &&
  python -m pip install --upgrade install/python/tensorrtserver-*.whl numpy pillow
12/19/2019 8:08 PM UTC
sha256:9de600edabf903dc07926c2fa39113e791db09181ebb91054b0ba3e8ed2a0bb3COPY
file:c8e3b9bf6aba2778f2250e95144b73aef129b32b070cb7eaf6148b53e4b19abe in images/mug.jpg
12/19/2019 8:07 PM UTC
sha256:9323a9bd970358e41f9cdc01e87fa49655f3870515cc789c6ea3b0b214bb230cCOPY
file:739b0ef33071ef274b805fc5b94589d10f3f0428805e32a8401e5b2d355eb58e in /tmp/test.sh
12/19/2019 8:07 PM UTC
sha256:76625f5900ef95dc984692c36611b2e8d0e4955811b7b5886ae49788c9d64402RUN
cd install &&
  export VERSION=`cat /workspace/VERSION` &&
  tar zcf /workspace/v$VERSION.clients.tar.gz *
12/19/2019 8:07 PM UTC
sha256:7e70000d898d9d01dcae30d119360391c04df10105c8e1cc0b1853188edd0d63RUN
cd build &&
  cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/workspace/install &&
  make -j16 trtis-clients
12/19/2019 8:07 PM UTC
sha256:b90a133124a8141b8b9960e51e39293ae9c45d5f10c2f566b81a8397fc89c562COPY
dir:acdb7f35ea8e714594327eb5600bd0aaa6cc1db56b38fd96ba9ac72d9eb373ea in src/core
12/19/2019 8:02 PM UTC
sha256:5989e4aa1f5486a7c675388e8045dd7a1d19f777216f68375f0730f250263381COPY
dir:9d2790409629cbebcddd6b512e2ad19414065b71bdd7d9f1f54b32e41d1dbcaa in src/clients
12/19/2019 8:02 PM UTC
sha256:14a3ab0e7e97d9d47786de62019b5811f405450b5d3357fa21d945185a60cfb4COPY
dir:7ff36ef21dd0ae4422703db989ca3712dc86e30af10ea756b344830ef69aafe4 in build
12/19/2019 8:02 PM UTC