ContainersNVIDIADeepSeek-R1NVIDIALlama-3.1-Nemotron-70B-InstructNVIDIAPyTorch
CollectionsNVIDIAOmniverse Kit (FB)NVIDIADeepStream SDKNVIDIAOmniverse Kit App Streaming
ModelsNVIDIAStyleGAN3 pretrained modelsNVIDIAPeopleNetNVIDIATrafficCamNet
ResourcesNVIDIARiva Skills Quick Start
Helm ChartsNVIDIAGPU OperatorNVIDIANVIDIA NIM Operator
NVIDIA
NVIDIA
TensorRT Inference Server
Container
NVIDIA
NVIDIA
TensorRT Inference Server

TensorRT Inference Server provides a data center inference solution optimized for NVIDIA GPUs. It maximizes inference utilization and performance on GPUs via an HTTP or gRPC endpoint, allowing remote clients to request inference for any model that is being managed by the server, as well as providing real-time metrics on latency and requests.

LayerLabelCreated
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4ENV
LD_LIBRARY_PATH=/workspace/install/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
02/18/2020 8:38 PM UTC
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4ENV
PATH=//workspace/install/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
02/18/2020 8:38 PM UTC
sha256:5dc5a0ec158f495c1d3a686317d2f0e26e18fa487c8e6ae1c2ffa1b5ffe8c2b4RUN
pip3 install --upgrade install/python/tensorrtserver-*.whl numpy pillow
02/18/2020 8:38 PM UTC
sha256:d053fdcbb817014c1e87d3378c14af730bbde90b3039ee1b5c4d9da974ca4cf1COPY
file:c8e3b9bf6aba2778f2250e95144b73aef129b32b070cb7eaf6148b53e4b19abe in images/mug.jpg
02/18/2020 8:38 PM UTC
sha256:c40653c9f7ad88634d95c1a0dda6ff1e1faf6204eb4382528ca400cc083c2bcbCOPY
file:739b0ef33071ef274b805fc5b94589d10f3f0428805e32a8401e5b2d355eb58e in /tmp/test.sh
02/18/2020 8:38 PM UTC
sha256:315a931ce0e2d65de8312733a4ae2b951f8aeb28bb188c68ba6b8dfc38fcce45RUN
cd install &&
  export VERSION=`cat /workspace/VERSION` &&
  tar zcf /workspace/v$VERSION.clients.tar.gz *
02/18/2020 8:38 PM UTC
sha256:0a07892a45f3ca9d7b7d5332e08123a3f6de102d9fda2285df1c1f2b8ae919adRUN
cd build &&
  cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/workspace/install &&
  make -j16 trtis-clients
02/18/2020 8:38 PM UTC
sha256:6e994abe54d0d675f8b48484953c90f1b682b1191dde00fe0d31f0a7e23103b6COPY
dir:06043e1a32eced9a694932137a8add978538f8dcff56599128130ff398f3f4b2 in src/core
02/18/2020 8:33 PM UTC
sha256:8ba5ee25cefbae6c3ed1542473e76e735bb80206a4be65afd08de36f23c63e00COPY
dir:50dd7ae9794a956f4973ff7e693dc4ba45b6cdf10dbd5066b7c4912d549d040e in src/clients
02/18/2020 8:33 PM UTC
sha256:1ab365b227f59e418c917ac7b5710ed38ed82f96c0b5a3070cabf429ac4bbb37COPY
dir:f8670490a47eae7ec04455f70e2ed0b24f4fd7028c09d9762fbb97453efc6ee9 in build
02/06/2020 7:59 PM UTC