System Render Mode uses the Windows PDF API from the Windows.Data.Pdf namespace. This allows the C1PdfViewer control to render each page as an image. Generally, System mode is considered safer since it more reliably renders unpredictable PDF content. It's also faster at presenting content to the user since there is less calculation for positioning content.
When creating an app that allows users to simply view any PDF file they may have on their machine, setting the RenderMode property to System will allow you to reliably render some formats, such as deflated JPG streams, custom color spaces, and halftones. System render mode also allows you to render other formats that aren't supported in Default render mode, such as soft masks, TIFF data, and DCTDecode/JPEG image encoding.
This mode is more reliable for PDF features, but lacks support for text select and search capability.