Struct NvencEncoder 

pub struct NvencEncoder { /* private fields */ }

A live NVENC H.264 encoder session with its CUDA context and interop resources.

One instance owns a CUDA context bound to a specific GPU plus an NVENC session and everything the three input paths need:

• Packed path: a pitched device buffer (input_device_ptr / input_pitch) registered and mapped as the NVENC input (registered_input_resource / mapped_input_buffer) in the byte order input_format names (re-registered in place when a source of the other order arrives), fed either by a host→device upload or by the copy arm of the dmabuf path.
• Raw planar path: a lazily-allocated NV12 / YUV444 device buffer (the nv12_* fields), created on first encode_raw.
• Zero-copy dmabuf path: dmabuf_cache memoizes each fd’s EGLImage → CUDA import, keyed by fd but validated against the buffer’s DmaBufIdentity so a recycled fd re-imports; an import whose plane NVENC can take as it is — pitch-linear memory or a packed 8-bit CUDA array — is registered in place (DmaBufInput::Direct) unless direct_dmabuf was switched off, anything else is copied into the packed input each frame.

bitstream_buffers is a small ring (current_buffer_idx cycles it) of output buffers. pinned_hosts maps each page-locked host upload source’s base pointer to its registered length, with a 0 length recording a failed registration so that address is never re-pinned. current_qp tracks the live ConstQP so a paint-over reconfigure is skipped when unchanged. encode_config and init_params are retained so in-place reconfigure can resubmit them. omit_stripe_headers drops the 10-byte wire header, and node_index is the effective CUDA device this session is bound to — a reuse across captures that now targets a different device must rebuild rather than reconfigure.

Implementations

impl NvencEncoder

pub fn new( settings: &RustCaptureSettings, egl_display: *const c_void, ) -> Result<Self, String>

Build a live NVENC session: bind CUDA to the target GPU, open and configure the H.264 encoder, and allocate its input and output buffers.

The sequence:

1. Load and negotiate: dlopen EGL / CUDA / NVENC, then nvenc_negotiate resolves the API version against the driver (set-once) before any struct is version-tagged. The multi-GPU GET_ATTACHED_IDS ioctl filter is installed after the NVIDIA libraries are loaded — so their GOTs can be patched — and before cuInit enumerates devices (a no-op unless a host GPU is hidden from this container). The three library Arcs are leaked once per process so the resolved function pointers stay valid for the program’s life.
2. Bind the device: cuInit, then bind by the render node’s PCI bus ID (encode_node_index, with auto <0 meaning device 0), falling back to CUDA device 0, and retain the device’s primary CUDA context — shared and refcounted across every session on that device rather than a fresh 100-300 MiB context each — pushing it current.
3. Allocate input: a pitched ARGB device buffer (cuMemAllocPitch, 16-byte element alignment) that hardware CSC turns into YUV.
4. Open the session and query caps: create the function-list instance, open the session with the negotiated apiVersion, and query nvEncGetEncodeCaps so init degrades rather than fails — a 4:4:4 request on a GPU without it drops to 4:2:0, and a capture beyond the encoder’s max dimensions returns Err so the caller falls back to software. Then pull a preset config (P4, ultra-low-latency); a failed preset lookup logs the driver’s error string and proceeds with the zeroed default rather than aborting.
5. Configure the stream (mutating the returned preset config, whose version word is re-stamped while its embedded rcParams keeps the version the preset fill set): High or High-4:4:4 profile; CBR (two-pass quarter-resolution for tighter per-frame rate adherence, VBV sizing, optional min/max QP clamps) or ConstQP; infinite GOP (gopLength / idrPeriod = 0xFFFFFFFF); zeroReorderDelay plus a bitstream-restriction VUI (max_num_reorder_frames=0) so no-reorder decoders don’t buffer; an explicit Annex-A level from nvenc_h264_level pinned from frame 1 so the level never bumps mid-stream; BT.709 VUI primaries and transfer for the sRGB source, with an SMPTE170M matrix and limited range to match the hardware ARGB CSC; chroma 4:2:0 or 4:4:4; repeated SPS/PPS; CABAC; no AUD; strict GOP target; and lookahead disabled for real-time latency.
6. Initialize with resize headroom: maxEncodeWidth / maxEncodeHeight are raised to at least 4096×2304 (the 5.2 ceiling) so reconfigure_resolution can grow in place, but never past the driver’s reported maximum; this costs ~290 MiB of device memory, so a failed init retries at the exact size (in-place resize then falls back to a rebuild).
7. Register, map, and buffer: register and map the packed input surface (as ARGB; set_input_format re-registers it for an RGBA source), and create a 4-deep ring of bitstream output buffers.

Every failure after the CUDA allocation unwinds the resources created so far — buffers, session, context — before returning Err. EGL is only needed by the zero-copy dmabuf path, so callers on the host-ARGB path pass a null egl_display. The retained init_params.encodeConfig raw pointer is nulled before the struct is returned (it points at a local config about to move); the reconfigure paths repoint it at self.encode_config when they resubmit.

pub fn reconfigure_resolution( &mut self, settings: &RustCaptureSettings, ) -> Result<(), String>

Resize the live session to settings in place, folding in the current rate / QP / fps, without tearing it down.

The NVENC session, CUDA context and bitstream buffers survive, so a resize costs a few milliseconds instead of a full rebuild. Flow:

1. Reject the unchangeable: a different encode device, a chroma-format flip (4:4:4), an RC-mode flip, or dimensions of zero or beyond the init-time maxEncode headroom all return Err so the caller rebuilds. Chroma and RC mode are read back from the live encode_config (the H.264 arm of the codec-config union is the one this encoder fills).
2. Release geometry-dependent state under the pushed CUDA context: unmap / unregister / free the packed input surface, the raw-plane buffer, every cached dmabuf import (with the NVENC registration a direct import holds), and every pinned host. The raw-plane buffer and dmabuf imports are re-created lazily by their encode paths; pinned hosts are dropped because the source shm segments are recreated on resize and may reuse the same base addresses.
3. Reconfigure the session: update the level for the new size, the CBR bitrate + VBV or the ConstQP, and the new dimensions / DAR / frame rate, then NvEncReconfigureEncoder with resetEncoder and forceIDR so the stream restarts cleanly at the new size. Driver rejection returns Err.
4. Reallocate the packed input at the new size and register + map it as init does, in the byte order the session was last fed.

On success the next encoded frame is a reset-RC IDR.

pub fn release_pinned_hosts(&mut self)

Drop every page-locked host registration, under the pushed CUDA context.

Called when the capture’s shm segments are recreated at unchanged dimensions: the new segments often reuse the old base addresses, so a stale registration would alias fresh memory. Subsequent uploads re-pin lazily. A 0-length entry marks a registration that failed and so is not unregistered.

pub fn reconfigure_rate(&mut self, settings: &RustCaptureSettings)

Apply a runtime rate-control / frame-rate change to the live session.

In CBR mode the target bitrate, max bitrate and VBV buffer size are updated (the VBV is ignored outside CBR); the target fps is updated in either mode. The session is reconfigured only when one of these actually changed — no forced IDR, no RC reset — so calling it every frame is cheap.

pub fn encode( &mut self, dmabuf: &Dmabuf, frame_number: u64, target_qp: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a dmabuf frame zero-copy, by importing it through EGL into CUDA and, where the driver allows, handing the mapped plane to NVENC as its input.

Applies any pending ConstQP change, then works under the pushed CUDA context:

1. Import once, cache by fd with an identity check: the cache is keyed by the dmabuf fd but each entry stores the buffer’s DmaBufIdentity; an entry whose identity no longer matches (a recycled fd) is released first. On a miss, build an EGLImageKHR from the dmabuf’s fd / offset / pitch / modifier, register it as a CUDA graphics resource, map it to a CUeglFrame, and settle how it feeds the encoder: a first plane that direct_plane accepts (and direct_dmabuf on) is registered with NVENC in place — a pitch-linear plane as a CUDA device pointer at its own pitch, a four-channel 8-bit CUDA array as a CUDA array — in the byte order the dmabuf fourcc names, and mapped once (DmaBufInput::Direct); any other plane, or a registration the driver refuses, takes DmaBufInput::Copy. The result is memoized so a recurring capture buffer pays the import cost only once. Each failure destroys what it created and pops the context.
2. Feed the encoder: a direct import is submitted as it is — no copy at all. A copy import is copied with cuMemcpy2DAsync on the default stream — the array plane or the pitch-linear plane, per frame_type — into the packed input surface, re-registered in the dmabuf’s byte order when it differs; NVENC processes its input on that same stream, so the copy is ordered before the encode without a host wait.
3. Submit via submit_frame, then pop the context.

The dmabuf fd is read out before the context is pushed so an early ? return cannot leave the CUDA context stack imbalanced.

pub fn encode_cpu_argb( &mut self, argb: &[u8], src_stride: usize, frame_number: u64, target_qp: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a host BGRA frame (B,G,R,A in memory, NVENC’s word-ordered ARGB — the layout an XShm grab or the pixman framebuffer yields) through encode_cpu_packed.

pub fn encode_cpu_packed( &mut self, pixels: &[u8], src_stride: usize, rgba_input: bool, frame_number: u64, target_qp: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a host packed-pixel frame by uploading it straight into the packed input surface, with no CPU-side colour conversion: NVENC’s hardware RGB→YUV conversion is fixed at BT.601 limited range, which is what the session VUI declares, and a CPU prepass to BT.709 would cost this path its copy-free property.

rgba_input names the byte order — false for B,G,R,A (X11 XShm, the pixman framebuffer), true for R,G,B,A (a GLES readback) — and the input surface is registered with NVENC in that order (ARGB / ABGR, re-registered in place when it changes), so both arrive at the hardware CSC untouched. src_stride is the source row stride in bytes (>= width*4). Steps, under the pushed CUDA context after any pending QP change:

1. Bounds-check the source against stride × (rows-1) + width*4, erroring rather than reading past a short buffer.
2. Pin the source once: unless pinning was disabled at init (PIXELFLUX_NVENC_PIN=0, read once into pin_uploads), page-lock each distinct source base address via pin_host_source so the upload is a direct DMA from the caller’s buffer instead of a pageable copy staged through a driver bounce buffer. The persistent, bounded shm / pool sources make this a one-time bounded cost.
3. Upload and submit: cuMemcpy2DAsync the rows into the input surface on the default stream honoring src_stride, then submit_frame. NVENC processes its input on that same stream, so the upload is ordered before the encode without a host wait, and the blocking bitstream lock inside submit_frame (or the stream sync on its error path) guarantees the upload has finished reading pixels by the time this returns — the caller may reuse the buffer immediately.

pub fn encode_raw( &mut self, raw_data: &[u8], frame_number: u64, target_qp: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a raw planar frame — NV12 (4:2:0) or YUV444 — uploaded host→device.

The planar-input counterpart to encode_cpu_argb, used when the caller has already produced YUV. The chroma format follows the session’s chromaFormatIDC (3 ⇒ YUV444, else NV12). Flow under the pushed CUDA context after any pending QP change:

1. Pin the source once (unless disabled at init): the caller reuses one planar buffer across frames, so page-locking its base via pin_host_source turns each upload into a direct pinned DMA instead of a pageable copy through a bounce buffer.
2. Lazily allocate the planar device buffer on first use: a pitched allocation tall enough for three full planes (YUV444) or Y plus half-height interleaved UV (NV12), registered and mapped with the matching buffer format.
3. Upload each plane with its own cuMemcpy2D. Every copy is bounds-checked against the host slice: the Y plane is required in full (a short buffer errors), and each chroma plane is copied only if its entire span — not merely its start offset — is present, so a truncated buffer never reads past its end.
4. Submit the mapped planar input via submit_frame.

Trait Implementations

impl Drop for NvencEncoder

Release every GPU resource the session holds, in the one teardown order the drivers tolerate, so nothing leaks and no still-referenced handle is ever freed out from under the driver.

The whole sequence runs with the owning CUDA context pushed current, because the cuMemFree / cuGraphicsUnregisterResource / cuMemHostUnregister calls each act on the current context — pop it first and the frees silently do nothing, leaking device memory. Within that, resources go inner-handle before the outer handle that owns it, since freeing an owner first orphans or faults on what still points into it: unmap the packed and raw-plane inputs before unregistering them, free their device buffers, destroy the bitstream buffers, and release every cached dmabuf import (release_dmabuf_import: its NVENC mapping and registration, then the CUDA resource and the EGLImage) — all session-owned — before the encoder session itself, and destroy that session before releasing the device’s primary CUDA context it was opened against (the retain is refcounted, so the context lives until the last session on that device releases it). The page-locked host sources are unpinned in the same pass, each only when its recorded length is non-zero (a 0 marks a registration that failed and so was never pinned).

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for NvencEncoder