Observing VPN clients
Most of the VPN protocols supported by OpenConnect are proprietary, undocumented, and in many cases overly-complex, perhaps intentionally obfuscated.
In order to add support for new protocols, and sometimes even to improve or update support for existing ones, it is often necessary to observe existing clients and servers in order to understand how they work.
Observing TLS/SSL connections
Modern VPN protocols almost always support a UDP-based transport for tunneled packets, e.g. DTLS for the Cisco AnyConnect protocol, or ESP for the GlobalProtect protocol. This is because TCP over TCP is very suboptimal in terms of performance. However, most VPN protocols also support TLS/SSL for connection initiation and as a fallback, due to its universal availability even in highly filtered or firewalled network environments. It is typically more straightforward and productive to start out by observing the TLS/SSL side of a new VPN protocol, and saving the UDP-based transport for later (See this discussion thread from the mailing list during the initial work on the GlobalProtect protocol.)
The content of a TLS/SSL connection is end-to-end encrypted and authenticated, and the session keys are normally kept only in memory by the applications using these connections. Therefore, special techniques are needed to decrypt traffic and observe the plaintext bytes-on-the-wire of any TLS-based communications protocol.
There are more-or-less 3 possible techniques for decrypting TLS-based communications and observing the plaintext. The first two shown here are quite simply, but not universally usable; the third is more complex but should be applicable to any TLS-based VPN protocol.
(1) Using SSLKEYLOGFILE
If your VPN client is based on a standard TLS library and does not disable this mechanism, it is likely the most straightforward way to decrypt TLS traffic.
Most TLS libraries (including OpenSSL, GnuTLS, and LibNSS) support an environment variable called SSLKEYLOGFILE, which will cause applications using those libraries to write "premaster secrets" for their TLS/DTLS sessions to a file. This makes it straightforward to automatically decrypt the traffic using the Wireshark network protocol analyzer. Basically, you should run the VPN client that you're observing with this environment variable set, run Wireshark on the same computer, and configure Wireshark to use the resulting log file; see Wireshark's TLS decryption documentation for details.
Note that some proprietary VPN clients disable the SSLKEYLOGFILE mechanism to prevent decryption of their traffic, but many developers overlook this especially in early releases of their client software.
(2) Using server's RSA private key
If you have access to the RSA private key of a VPN server supporting the protocol you're observing (likely because you administer such a VPN), and can use RSA key exchange, then you can provide this RSA private key to Wireshark and automatically decrypt TLS traffic to and from this server using Wireshark; again, see Wireshark's TLS decryption documentation for details and limitations.
This works because the RSA key exchange supported by TLS (re)uses the server's private key for encryption of th