
[[networking]]
= Networking in openQA
:toc: left
:toclevels: 6
:author: openQA Team

For tests using the QEMU backend the networking type used is controlled by the
`NICTYPE` variable. If unset or empty `NICTYPE` defaults to `user`, i.e.
<<QEMU User Networking>> which requires no further configuration.

For more advanced setups or tests that require multiple jobs to be in the same
networking the <<TAP based network,TAP>> or <<VDE Based Network,VDE>> based
modes can be used.

Other backends can be treated just the same as bare-metal setups. Tests can be
triggered in parallel same as for QEMU based ones and synchronization
primitives can be used. For the physical network according separation needs to
be ensured externally where needed as means for machines to be able to access
each other.

== QEMU User Networking
:qemu-user-networking: http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29[user networking]

With QEMU {qemu-user-networking} each jobs gets its own isolated network with
TCP and UDP routed to the outside. DHCP is provided by QEMU. The MAC address of
the machine can be controlled with the `NICMAC` variable. If not set, it is
`52:54:00:12:34:56`.

== TAP Based Network

os-autoinst can connect QEMU to TAP devices of the host system to
leverage advanced network setups provided by the host by setting `NICTYPE=tap`.

The TAP device to use can be configured with the `TAPDEV` variable. If not
defined, it is automatically set to "tap" + ($worker_instance - 1), i.e.
worker1 uses tap0, worker 2 uses tap1 and so on.

For multiple networks per job (see `NETWORKS` variable), the following numbering
scheme is used:

[source,sh]
----
worker1: tap0 tap64 tap128 ...
worker2: tap1 tap65 tap129 ...
worker3: tap2 tap66 tap130 ...
...
----

The MAC address of each virtual NIC is controlled by the `NICMAC` variable or
automatically computed from `$worker_id` if not set.

In TAP mode the system administrator is expected to configure the network,
required internet access, etc. on the host as described in the next section.


=== Multi-machine test setup

The section provides one of the ways for setting up the openQA environment to
run tests that require network connection between several machines (e.g.
client -- server tests).

The example of the configuration is applicable for openSUSE and will use _Open
vSwitch_ for virtual switch, _firewalld_ (or _SuSEfirewall2_ for older
versions) for NAT and _wicked_ as network manager. Keep in mind that a
firewall is not strictly necessary for operation. The operation without
firewall is not covered in all necessary details in this documentation.

NOTE: Another way to setup the environment with _iptables_ and _firewalld_ is described
on the link:https://fedoraproject.org/wiki/OpenQA_advanced_network_guide[Fedora wiki].

NOTE: The salt states contained by
https://github.com/os-autoinst/salt-states-openqa[salt-states-openqa] contain all the
necessities to establish such a setup and configure it for all workers with the `tap`
worker class. They also cover GRE tunnels (that are explained in the next section).

*Set Up Open vSwitch*

Compared to VDE setup, Open vSwitch is slightly more complicated to configure,
but provides a more robust and scalable network.

* Install and Run Open vSwitch:

[source,sh]
----
zypper in openvswitch
systemctl enable --now openvswitch
----

*  Install and configure _os-autoinst-openvswitch.service_:

NOTE: _os-autoinst-openvswitch.service_ is a support service that sets the
vlan number of Open vSwitch ports based on `NICVLAN` variable - this separates
the groups of tests from each other. The `NICVLAN` variable is dynamically
assigned by the openQA scheduler. Install, start and enable the service:

[source,sh]
----
zypper in os-autoinst-openvswitch
systemctl enable --now os-autoinst-openvswitch
----

The service _os-autoinst-openvswitch.service_ uses _br0_ bridge by default.
As it might be used by KVM already it is suggested to configure _br1_ instead:

[source,sh]
----
# /etc/sysconfig/os-autoinst-openvswitch
OS_AUTOINST_USE_BRIDGE=br1
----

* Create the virtual bridge _br1_:
[source,sh]
----
ovs-vsctl add-br br1
----

==== Configure virtual interfaces

* Add a tap interface for every multi-machine worker instance:

NOTE: Create as many interfaces as needed for a test. The instructions are
provided for three interfaces _tap0_, _tap1_, _tap2_ to be used by _worker@1_,
_worker@2_, _worker@3_ worker instances. The TAP interfaces have to be owned
by the __openqa-worker_ user for the openQA worker instances to be able to
access them.

To create tap interfaces automatically on startup, add appropriate configuration files to the
`/etc/sysconfig/network/` directory. Files have to be named as `ifcfg-tap<N>`, replacing `<N>`
with the number for the interface, such as `0`, `1`, `2` (e.g. `ifcfg-tap0`,
`ifcfg-tap1`):

[source,sh]
----
# /etc/sysconfig/network/ifcfg-tap0
BOOTPROTO='none'
IPADDR=''
NETMASK=''
PREFIXLEN=''
STARTMODE='auto'
TUNNEL='tap'
TUNNEL_SET_GROUP='nogroup'
TUNNEL_SET_OWNER='_openqa-worker'
----

Symlinks can be used to reference the same configuration file for each tap
interface.

* Add the bridge config with all tap devices that should be connected to it.
  The file has to be located in the `/etc/sysconfig/network/` directory. File
  name is `ifcfg-br<N>`, where `<N>` is the id of the bridge (e.g. `1`):

[source,sh]
----
# /etc/sysconfig/network/ifcfg-br1
BOOTPROTO='static'
IPADDR='10.0.2.2/15'
STARTMODE='auto'
OVS_BRIDGE='yes'
OVS_BRIDGE_PORT_DEVICE_1='tap0'
OVS_BRIDGE_PORT_DEVICE_2='tap1'
OVS_BRIDGE_PORT_DEVICE_3='tap2'
----

==== Configure NAT with firewalld
You can just create a configuration file for the `trusted` zone like this:
[source,sh]
----
# /etc/firewalld/zones/trusted.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <short>Trusted</short>
  <description>All network connections are accepted.</description>
  <interface name="br1"/>
  <interface name="ovs-system"/>
  <interface name="eth0"/>
  <masquerade/>
</zone>
----

It is important that masquerading is enabled. Then set the default zone accordingly
in `/etc/firewalld/firewalld.conf`, e.g. `DefaultZone=trusted`. The changes will be
effective on the next reboot which can be checked by invoking e.g.
`firewall-cmd --zone=trusted --list-all`. It should show the configured interfaces
and `masquerade: yes`. The firewall is configured this way by the previously
mentioned salt states.

You can of course also use `firewall-cmd` manually and use different zones. For
instance, you can assign the bridge interface to the internal zone and the interface
with access to the network to the external zone:

[source,sh]
----
firewall-cmd --zone=external --add-interface=eth0
firewall-cmd --zone=internal --add-interface=br1
----

To enable the virtual machines used by openQA to fully access the external
network masquerading needs to be enabled on all involved zones:

[source,sh]
----
firewall-cmd --zone=external --add-masquerade
firewall-cmd --zone=internal --add-masquerade
----

IP forwarding is enabled automatically if masquerading is enabled:

[source,sh]
----
grep 1 /proc/sys/net/ipv4/ip_forward
1
----

In case the interface is in a trusted network it is possible to accept
connections by default by changing the zone target:

[source,sh]
----
firewall-cmd --zone=external --set-target=ACCEPT
----

If you are happy with the changes make them persistent:

[source,sh]
----
firewall-cmd --runtime-to-permanent
----

If you do not currently have the firewalld service running, you can instead
use the `firewall-offline-cmd` command for the configuration. In this case
start the firewall and enable the service to run on system startup:

[source,sh]
----
systemctl enable --now firewalld
----

Also, the `firewall-config` GUI tool for firewalld can be used for configuration.

==== For older versions of openSUSE/SLE: Configure NAT with SuSEfirewall2

The IP 10.0.2.2 can be also served as a gateway to access the outside network.
For this, NAT between _br1_ and _eth0_ must be configured with SuSEfirewall2
or iptables:

[source,sh]
----
# /etc/sysconfig/SuSEfirewall2
FW_DEV_INT="br1"
FW_ROUTE="yes"
FW_MASQUERADE="yes"
----

Start SuSEfirewall2 and enable the service to start on system startup:

[source,sh]
----
systemctl enable --now SuSEfirewall2
----


==== Configure openQA workers
Allow worker intstances to run multi-machine jobs:

[source,sh]
----
# /etc/openqa/workers.ini
[global]
WORKER_CLASS = qemu_x86_64,tap
----

NOTE: The number of tap devices should correspond to the number of the running
worker instances. For example, if you have set up 3 tap devices, the same
number of worker instances should be configured.

Enable worker instances to be started on system boot:

[source,sh]
----
systemctl enable openqa-worker@1
systemctl enable openqa-worker@2
systemctl enable openqa-worker@3
----

==== Grant CAP_NET_ADMIN Capabilities to QEMU
In order to let QEMU create TAP devices on demand it is required to set
CAP_NET_ADMIN capability on QEMU binary file:

[source,sh]
----
zypper in libcap-progs
setcap CAP_NET_ADMIN=ep /usr/bin/qemu-system-x86_64
----

==== Configure network interfaces
Check the configuration for the _eth0_ interface:

IMPORTANT: Ensure, that _eth0_ interface is configured in
`/etc/sysconfig/network/ifcfg-eth0`. Otherwise, wicked will not be able to
bring up the interface on start and the host will loose network connection:

[source,sh]
----
# /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='dhcp'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
REMOTE_IPADDR=''
STARTMODE='auto'
DHCLIENT_SET_DEFAULT_ROUTE='yes'
----

Pros of wicked over NetworkManager:

** Proper IPv6 support
** openvswitch/vlan/bonding/bridge support - wicked can manage your advanced configuration transparently without the need of extra tools
** Backwards compatible with ifup scripts

Check the network service currently being used:

[source,sh]
----
systemctl show -p Id network.service
----

If the result is different from `Id=wicked.service` (e.g.
`NetworkManager.service`), stop the network service:

[source,sh]
----
systemctl disable --now network.service
----

Then switch to wicked and start the service:

[source,sh]
----
systemctl enable --force wicked
systemctl start wicked
----

Bring up the _br1_ interface:

[source,sh]
----
wicked ifup br1
----

Finally, reboot the system.

NOTE: It is also possible to switch the network configuration using YaST.

=== GRE tunnels
By default all multi-machine workers have to be on a single physical machine.
You can join multiple physical machines and its OVS bridges together by a GRE
tunnel.

If the workers with TAP capability are spread across multiple hosts, the
network must be connected. See Open vSwitch
http://openvswitch.org/support/config-cookbooks/port-tunneling/[documentation]
for details.

Create a gre_tunnel_preup script (change the `remote_ip` value correspondingly
on both hosts):

[source,sh]
----
# /etc/wicked/scripts/gre_tunnel_preup.sh
#!/bin/sh
action="$1"
bridge="$2"
ovs-vsctl set bridge $bridge stp_enable=true
ovs-vsctl --may-exist add-port $bridge gre1 -- set interface gre1 type=gre options:remote_ip=<IP address of other host>
----

And call it by PRE_UP_SCRIPT="wicked:gre_tunnel_preup.sh" entry:

[source,sh]
----
# /etc/sysconfig/network/ifcfg-br1
<..>
PRE_UP_SCRIPT="wicked:gre_tunnel_preup.sh"
----

Ensure to make gre_tunnel_preup.sh executable.

Allow GRE in older setups still using SuSEfirewall2:

[source,sh]
----
# /etc/sysconfig/SuSEfirewall2
FW_SERVICES_EXT_IP="GRE"
FW_SERVICES_EXT_TCP="1723"
----

NOTE: When using GRE tunnels keep in mind that virtual machines inside the ovs
bridges have to use MTU=1458 for their physical interfaces (eth0, eth1). If
you are using support_server/setup.pm the MTU will be set automatically to
that value on support_server itself and it does MTU advertisement for DHCP
clients as well.

=== Verify the setup
Simply run a MM test scenario. For openSUSE, you can find many relevant tests
on https://openqa.opensuse.org[o3], e.g. look for networking-related tests like
wicked-tests. To test GRE tunnels, you may want to change the jobs worker classes
so the different jobs are executed on different workers.

So you could call `openqa-clone-job` like this:
```
openqa-clone-job \
    --skip-download --skip-chained-deps \    # assuming assets are present
    --max-depth 0 \                          # clone the entire parallel cluster
    --export-command \                       # only print the API call
    https://openqa.opensuse.org/tests/250309 # arbitrary job in cluster to clone
    _GROUP=0 BUILD+=test-mm-setup            # avoid interfering with production jobs
```

It will print an `openqa-cli` call. You can modify it to change the worker classes of
the jobs individually and then invoke it.

Also be sure to reboot the worker host to make sure the setup is actually persistent.

=== Debugging Open vSwitch Configuration
Boot sequence with wicked (version 0.6.23 and newer):

1. openvswitch (as above)
2. wicked - creates the bridge `br1` and tap devices, adds tap devices to the bridge,
3. firewalld (or SuSEfirewall2 in older setups)
4. os-autoinst-openvswitch - installs openflow rules, handles vlan assignment


The configuration and operation can be checked with the following commands:

[source,sh]
----
ovs-vsctl show # shows the bridge br1, the tap devices are assigned to it
ovs-ofctl dump-flows br1 # shows the rules installed by os-autoinst-openvswitch in table=0
ovs-dpctl show # show basic info on all datapaths
ovs-dpctl dump-flows # displays flows in datapaths
----

When everything is ok and the machines are able to communicate, the ovs-vsctl
should show something like the following:

[source,sh]
----
Bridge "br0"
    Port "br0"
        Interface "br0"
            type: internal
    Port "tap0"
        Interface "tap0"
    Port "tap1"
        tag: 1
        Interface "tap1"
    Port "tap2"
        tag: 1
        Interface "tap2"
  ovs_version: "2.11.1"
----

NOTE: Notice the tag numbers are assigned to tap1 and tap2. They should have
the same number.

NOTE: If the balance of the tap devices is wrong in the workers.ini the tag
cannot be assigned and the communication will be broken.

To list the rules which are effectively configured in the underlying netfilter
(`nftables` or `iptables`) use one of the following commands depending on which
netfilter is used.

NOTE: Whether firewalld is using `nftables` or `iptables` is determined by the
setting `FirewallBackend` in `/etc/firewalld/firewalld.conf`. SuSEfirewall2 is
always using `iptables`.

[source,sh]
----
nft list tables           # list all tables
nft list table firewalld  # list all rules in the specified table
----

[source,sh]
----
iptables --list --verbose # list all rules with package counts
----

Check the flow of packets over the network:

* packets from tapX to br1 create additional rules in table=1
* packets from br1 to tapX increase packet counts in table=1
* empty output indicates a problem with os-autoinst-openvswitch service
* zero packet count or missing rules in table=1 indicate problem with tap devices

As long as the SUT has access to external network, there should be a non-zero
packet count in the forward chain between the br1 and external interface.

NOTE: To list the package count when `nftables` is used one needed to use
https://wiki.nftables.org/wiki-nftables/index.php/Counters[counters] (which can
be https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/getting-started-with-nftables_configuring-and-managing-networking#adding-a-counter-to-an-existing-rule_debugging-nftables-rules[added to existing rules]).


== VDE Based Network

Virtual Distributed Ethernet provides a software switch that runs in
user space. It allows to connect several QEMU instances without
affecting the system's network configuration.

The openQA workers need a vde_switch instance running. The workers
reconfigure the switch as needed by the job.

=== Basic, Single Machine Tests

To start with a basic configuration like QEMU user mode networking,
create a machine with the following settings:

- `VDE_SOCKETDIR=/run/openqa`
- `NICTYPE=vde`
- `NICVLAN=0`

Start the switch and user mode networking:

[source,sh]
----
systemctl enable --now openqa-vde_switch
systemctl enable --now openqa-slirpvde
----

With this setting all jobs on the same host would be in the same network and
share the same SLIRP instance.
