Struct NvencEncoder 

pub struct NvencEncoder { /* private fields */ }

A live NVENC encoder session with its CUDA context and interop resources.

One instance owns a CUDA context bound to a specific GPU plus an NVENC session and everything the two input paths need:

• Packed path: a pitched device buffer (input_device_ptr / input_pitch) registered and mapped as the NVENC input (registered_input_resource / mapped_input_buffer) in the byte order input_format names (re-registered in place when a source of the other order arrives), fed either by a host→device upload or by the copy arm of the dmabuf path.
• Zero-copy dmabuf path: dmabuf_cache memoizes each fd’s EGLImage → CUDA import, keyed by fd but validated against the buffer’s DmaBufIdentity so a recycled fd re-imports; an import whose plane NVENC can take as it is — pitch-linear memory or a packed 8-bit CUDA array — is registered in place (DmaBufInput::Direct) unless direct_dmabuf was switched off, anything else is copied into the packed input each frame.

bitstream_buffers is a ring of BITSTREAM_BUFFERS output buffers (current_buffer_idx cycles it). pinned_hosts maps each page-locked host upload source’s base pointer to its registered length, with a 0 length recording a failed registration so that address is never re-pinned. codec and fullcolor name the session’s codec and negotiated chroma; current_qp tracks the live ConstQP so a paint-over reconfigure is skipped when unchanged. encode_config and init_params are retained so in-place reconfigure can resubmit them. omit_stripe_headers drops the wire header, and node_index is the effective CUDA device this session is bound to — a reuse across captures that now targets a different device must rebuild rather than reconfigure.

Implementations

impl NvencEncoder

pub fn new( settings: &RustCaptureSettings, egl_display: *const c_void, ) -> Result<Self, String>

Build a live NVENC session for the settings’ codec: bind CUDA to the target GPU, open and configure the encoder, and allocate its input and output buffers.

The sequence:

1. Load and negotiate: dlopen EGL / CUDA / NVENC, then nvenc_negotiate resolves the API version against the driver (set-once) before any struct is version-tagged. The multi-GPU GET_ATTACHED_IDS ioctl filter is installed after the NVIDIA libraries are loaded — so their GOTs can be patched — and before cuInit enumerates devices (a no-op unless a host GPU is hidden from this container). The three library Arcs are leaked once per process so the resolved function pointers stay valid for the program’s life.
2. Bind the device: cuInit, then bind by the render node’s PCI bus ID (encode_node_index, with auto <0 meaning device 0), falling back to CUDA device 0, and retain the device’s primary CUDA context — shared and refcounted across every session on that device rather than a fresh 100-300 MiB context each — pushing it current.
3. Allocate input: a pitched ARGB device buffer (cuMemAllocPitch, 16-byte element alignment) that the chroma convert, or hardware CSC, turns into YUV.
4. Open the session and query caps: create the function-list instance, open the session with the negotiated apiVersion, refuse a codec the device lists no engine for, and query nvEncGetEncodeCaps so init degrades rather than fails — a 4:4:4 request on a GPU or codec without it drops to 4:2:0, and a capture beyond the encoder’s max dimensions returns Err so the caller falls back to software. Then pull a preset config (P3, ultra-low-latency); a failed preset lookup logs the driver’s error string and proceeds with the zeroed default rather than aborting.
5. Configure the stream (mutating the returned preset config, whose version word is re-stamped while its embedded rcParams keeps the version the preset fill set): the codec’s 4:2:0 or 4:4:4 profile; CBR (two-pass quarter-resolution for tighter per-frame rate adherence, VBV sizing, optional min/max QP clamps in the codec’s quantizer domain) or ConstQP; infinite GOP (gopLength / idrPeriod = 0xFFFFFFFF); zeroReorderDelay and, for H.264, a bitstream-restriction VUI (max_num_reorder_frames=0) so no-reorder decoders don’t buffer; an explicit level from nvenc_level pinned from frame 1 so the level never bumps mid-stream; BT.709 primaries and transfer for the sRGB source, at limited range, with the matrix whichever conversion the session uses produces; a decoded picture buffer of as many frames as the level admits, so a frame a client lost can be left out of the predictions (invalidate_reference) with earlier frames still there to predict from; repeated parameter sets on every key frame; H.264 CABAC; no AUD; one AV1 tile; strict GOP target; and lookahead disabled for real-time latency.
6. Initialize with resize headroom: maxEncodeWidth / maxEncodeHeight are raised to at least HEADROOM_WIDTH x HEADROOM_HEIGHT so reconfigure_resolution can grow in place, but never past the driver’s reported maximum; this costs a few hundred MiB of device memory, so a failed init reopens the session and retries at the exact size (in-place resize then falls back to a rebuild).
7. Register, map, and buffer: register and map the packed input surface (as ARGB; set_input_format re-registers it for an RGBA source), and create a 4-deep ring of bitstream output buffers.

Every failure after the CUDA allocation unwinds the resources created so far — buffers, session, context — before returning Err. EGL is only needed by the zero-copy dmabuf path, so callers on the host-ARGB path pass a null egl_display. The retained init_params.encodeConfig raw pointer is nulled before the struct is returned (it points at a local config about to move); the reconfigure paths repoint it at self.encode_config when they resubmit.

pub fn last_reference(&self) -> Reference

The frame the last encoded frame predicted from.

pub fn invalidate_reference(&mut self, frame_id: u16) -> bool

Leave frame frame_id and every frame after it out of the predictions. False when the device cannot or refuses, and the caller codes a key frame instead.

pub fn acknowledge_reference(&mut self, frame_id: u16, held: bool)

Every consumer holds frame frame_id, or where not held was sent it (ReferenceWindow::acknowledge).

pub fn take_report(&mut self, report: ReferenceReport) -> bool

Apply what the consumers say of a frame (FrameEncoder::take_report); false where a key frame is coded instead.

pub fn split_summary(&self) -> String

How the session splits a frame across the device’s encode engines (split_mode), for the line that says which device encodes.

pub fn device_name(&self) -> &str

The codec the session emits. The name CUDA gives the GPU this session encodes on.

pub fn codec(&self) -> Codec

pub fn is_full_range(&self) -> bool

Every NVENC session converts to and declares limited range.

pub fn is_fullcolor(&self) -> bool

Whether the session negotiated 4:4:4 chroma.

pub fn bit_depth(&self) -> u32

The bits per sample the session codes.

pub fn reconfigure_resolution( &mut self, settings: &RustCaptureSettings, ) -> Result<bool, String>

Follow a capture restart on the live session, folding in the current rate / QP / fps, without tearing it down.

The NVENC session, CUDA context, and bitstream buffers survive, so a restart costs a few milliseconds instead of a full rebuild. Flow:

1. Reject the unchangeable: a different encode device or codec, a chroma-format flip (4:4:4), an RC-mode flip, or dimensions of zero or beyond the init-time maxEncode headroom all return Err so the caller rebuilds.
2. Keep the stream at unchanged dimensions: the reference chain, the input surface, and the dmabuf imports stay as they are, so the restart costs no IDR and no reset. Only the pinned hosts are dropped – the restart recreates the source buffers, often at the same addresses – and the rate, frame rate, and wire framing the restart carries are folded in, as reconfigure_rate does. Returns Ok(false).
3. Release geometry-dependent state under the pushed CUDA context: unmap / unregister / free the packed input surface, every cached dmabuf import (with the NVENC registration a direct import holds), and every pinned host. The dmabuf imports are re-created lazily by the encode path; pinned hosts are dropped because the source shm segments are recreated on resize and may reuse the same base addresses.
4. Reconfigure the session: update the level for the new size, the decoded picture buffer where that level admits fewer frames (the driver takes a smaller one at the forced IDR and refuses to raise it again), the CBR bitrate + VBV or the ConstQP, and the new dimensions / DAR / frame rate, then NvEncReconfigureEncoder with resetEncoder and forceIDR so the stream restarts cleanly at the new size. Driver rejection returns Err.
5. Reallocate the packed input at the new size and register + map it as init does, in the byte order the session was last fed.

On a resize the next encoded frame is a reset-RC IDR and Ok(true) is returned.

pub fn release_pinned_hosts(&mut self)

Drop every page-locked host registration, under the pushed CUDA context.

Called when the capture’s shm segments are recreated at unchanged dimensions: the new segments often reuse the old base addresses, so a stale registration would alias fresh memory. Subsequent uploads re-pin lazily. A 0-length entry marks a registration that failed and so is not unregistered.

pub fn last_quality(&self) -> Option<u32>

The quality index the rate control last coded a frame at, where the driver reports its quantizer; a held frame (hold_quantizer) is not the rate control’s and leaves it.

pub fn last_size(&self) -> Option<usize>

The bytes of the last frame the rate control coded; a held frame leaves it.

pub fn hold_quantizer(&mut self, crf: u32, band: Option<(f64, f64)>)

Encode the next frame at the constant quantizer the quality index crf selects, whatever the rate control, and leave the session’s own rate control and quantizer as they were for the frame after: the cleanup of a still screen. A held key frame of a constant-rate session that comes out past HELD_KEY_BUDGET_S of the target is coded again, as a key frame, at the coarser quantizer held_key_retry picks, and so is a held frame of the whole picture past HELD_REFRESH_LIMIT_S, predicted from the frame before the first attempt, which the device leaves out as it does a frame the client lost (retract_last). band, the share of the picture from and to in raster order, confines crf to the blocks it covers (band_size), the rest of the frame held at the coarsest quantizer, or for AV1 at the coarsest the delta map’s 128 steps reach above the band: a still region is left as it is at any quantizer, and a change the frame carries before its damage is known (X11 under Turbo hashes a frame beside its encode) costs what the rate control’s own frame would.

pub fn band_size(&self) -> Option<usize>

The bytes of the last held frame (0 before one), where the session holds a band: H.264, HEVC and AV1, whose QP delta map is one entry a macroblock, a 32x32 coding tree block, or a 64x64 superblock.

pub fn reconfigure_rate(&mut self, settings: &RustCaptureSettings) -> bool

Apply a runtime rate-control / frame-rate change to the live session, and report whether the session carries it afterwards.

In CBR mode the target bitrate, max bitrate, VBV, and its initial delay are updated (the VBV is ignored outside CBR); the target fps is updated in either mode. The session is reconfigured only when one of these actually changed — no RC reset, and a forced IDR only where a target past the declared level’s bitrate ceiling, or a frame rate past its macroblock rate, raises the level, which the decoder learns from the sequence header a key frame carries; a level a lower target or frame rate no longer needs stays, since a level only ever has to be high enough and the driver refuses one below the decoded picture buffer the session declared — so calling it every frame is cheap. A reconfigure the driver refuses leaves the session encoding at its previous rate, logged with the driver’s reason.

pub fn encode( &mut self, dmabuf: &Dmabuf, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a dmabuf frame zero-copy, by importing it through EGL into CUDA and, where the driver allows, handing the mapped plane to NVENC as its input.

Applies any pending ConstQP change, then works under the pushed CUDA context:

1. Import once, cache by fd with an identity check: the cache is keyed by the dmabuf fd but each entry stores the buffer’s DmaBufIdentity; an entry whose identity no longer matches (a recycled fd) is released first. On a miss, build an EGLImageKHR from the dmabuf’s fd / offset / pitch / modifier, register it as a CUDA graphics resource, map it to a CUeglFrame, and settle how it feeds the encoder: a first plane that direct_plane accepts (and direct_dmabuf on) is registered with NVENC in place — a pitch-linear plane as a CUDA device pointer at its own pitch, a four-channel 8-bit CUDA array as a CUDA array — in the byte order the dmabuf fourcc names, and mapped once (DmaBufInput::Direct); any other plane, or a registration the driver refuses, takes DmaBufInput::Copy. The result is memoized so a recurring capture buffer pays the import cost only once. Each failure destroys what it created and pops the context.
2. Feed the encoder: a direct import is submitted as it is — no copy at all. A copy import is copied with cuMemcpy2DAsync on the default stream — the array plane or the pitch-linear plane, per frame_type — into the packed input surface, re-registered in the dmabuf’s byte order when it differs; NVENC processes its input on that same stream, so the copy is ordered before the encode without a host wait.
3. Submit via submit_frame, then pop the context.

The dmabuf fd is read out before the context is pushed so an early ? return cannot leave the CUDA context stack imbalanced.

pub fn encode_after_blit( &mut self, dmabuf: &Dmabuf, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

encode, for a buffer the X server is still blitting into: the frame’s work is queued behind a wait on the semaphore the server signals once the blit has landed (blit_semaphore_fd), so it runs on the GPU right behind the blit instead of after the CPU has learnt of it. NVENC reads its input on the default stream, as the copy and the convert do, so one wait there orders all of it.

pub fn blit_semaphore_fd(&mut self) -> Result<OwnedFd, String>

A new semaphore for the X server to signal after a blit and encode_after_blit to wait on: made on this session’s GPU, imported into its context in place of any earlier one, and returned as the fd the server imports.

pub fn encode_cpu_argb( &mut self, argb: &[u8], src_stride: usize, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a host BGRA frame (B,G,R,A in memory, NVENC’s word-ordered ARGB — the layout an XShm grab or the pixman framebuffer yields) through encode_cpu_packed.

pub fn encode_cpu_packed( &mut self, pixels: &[u8], src_stride: usize, rgba_input: bool, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a host packed-pixel frame by uploading it straight into the packed input surface, with no CPU-side color conversion: the surface is either the chroma convert’s source or, where the kernel did not load, NVENC’s own conversion’s; a host prepass would cost this path its copy-free property.

rgba_input names the byte order — false for B,G,R,A (X11 XShm, the pixman framebuffer), true for R,G,B,A (a GLES readback) — and the input surface is registered with NVENC in that order (ARGB / ABGR, re-registered in place when it changes), so both arrive at the hardware CSC untouched. src_stride is the source row stride in bytes (>= width*4). Steps, under the pushed CUDA context after any pending QP change:

1. Bounds-check the source against stride × (rows-1) + width*4, erroring rather than reading past a short buffer.
2. Pin the source once: unless pinning was disabled at init (PIXELFLUX_NVENC_PIN=0, read once into pin_uploads), page-lock each distinct source base address via pin_host_source so the upload is a direct DMA from the caller’s buffer instead of a pageable copy staged through a driver bounce buffer. The persistent, bounded shm / pool sources make this a one-time bounded cost.
3. Upload and submit: cuMemcpy2DAsync the rows into the input surface on the default stream honoring src_stride, then submit_frame. NVENC processes its input on that same stream, so the upload is ordered before the encode without a host wait, and the blocking bitstream lock inside submit_frame (or the stream sync on its error path) guarantees the upload has finished reading pixels by the time this returns — the caller may reuse the buffer immediately.

pub fn width(&self) -> u32

The geometry the session is currently initialized for.

pub fn height(&self) -> u32

The geometry the session is currently initialized for.

pub fn release_external_input(&mut self)

Drop the registration of an externally-owned input surface, under the pushed CUDA context.

The caller owns the memory behind it, so this must run before that owner releases it: a registration outliving its buffer leaves the driver holding a mapping of freed video memory.

pub fn encode_cuda_pitch( &mut self, device_ptr: CUdeviceptr, pitch: usize, rgba: bool, frame_number: u64, crf: u32, force_idr: bool, ) -> Result<Vec<u8>, String>

Encode a frame that already lives in video memory, reading it exactly where it was produced.

This is the zero-copy hand-over: device_ptr addresses packed pixels in this session’s own CUDA context — the X11 NvFBC capture buffer the NVIDIA driver composited the screen into — so no upload or cuMemcpy stands between the screen and the bitstream. The chroma convert reads that buffer where it lies, as it does for every other input, and a session without one hands the packed surface to NVENC’s own conversion instead, registered in place. rgba names the byte order (false for the B,G,R,A the driver’s native format delivers), and pitch is the buffer’s row stride in bytes.

The caller must keep the buffer alive and unmodified until this returns; the blocking bitstream lock inside means the encoder has finished reading by then, so the next capture may overwrite it.

pub fn hash_next_upload(&mut self, rows: u32) -> bool

Hash the next host frame’s bands of rows rows on the device as it encodes, for take_upload_hashes to read after: the change detection a Turbo frame’s paint-over reads, for the cost of a kernel beside the encode instead of a CPU pass over the frame. False where the driver refuses the kernel, which leaves the hash to the caller.

pub fn take_upload_hashes(&mut self) -> Option<Vec<u64>>

The band hashes of the host frame last coded where hash_next_upload asked for them.

pub fn band_hashes( &mut self, device_ptr: CUdeviceptr, pitch: usize, width: u32, height: u32, rows: u32, ) -> Option<Vec<u64>>

The hash of each band of rows rows of the widthxheight packed frame at device_ptr (pitch bytes a row, in this session’s context), computed where the frame lies: the change detection of a capture that reports none, NvFBC’s. None where the driver refuses the kernel.

Trait Implementations

impl Drop for NvencEncoder

Release every GPU resource the session holds, in the one teardown order the drivers tolerate, so nothing leaks and no still-referenced handle is ever freed out from under the driver.

The whole sequence runs with the owning CUDA context pushed current, because the cuMemFree / cuGraphicsUnregisterResource / cuMemHostUnregister calls each act on the current context — pop it first and the frees silently do nothing, leaking device memory. Within that, resources go inner-handle before the outer handle that owns it, since freeing an owner first orphans or faults on what still points into it: unmap the packed input before unregistering it, free its device buffer, destroy the bitstream buffers, and release every cached dmabuf import (release_dmabuf_import: its NVENC mapping and registration, then the CUDA resource and the EGLImage) — all session-owned — before the encoder session itself, and destroy that session before releasing the device’s primary CUDA context it was opened against (the retain is refcounted, so the context lives until the last session on that device releases it). The page-locked host sources are unpinned in the same pass, each only when its recorded length is non-zero (a 0 marks a registration that failed and so was never pinned).

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for NvencEncoder