Linux Interface bonding

interface bonding
(reference - https://wiki.linuxfoundation.org/networking/bonding)

purpose - to provide always connected / seamless connectivity service

The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical bonded interface
 => source code available in drivers/net/bonding directory
bonding driver creates a logical network interface by using multiple physical network interfaces underneath

Bonding provides benefits such as link aggregation (bandwidth), redundancy (availability) etc. 
Upper layers communicate through the logical bond interface which has an IP address but eventually the active physical interface(s) communicate thus hiding the actual interface.

Permanent Address of physical link (virtual link does not have it)
command - ethtool -P eth01
Permanent address: 2c:59:77:95:22:11

The real/ physical port/ interface has Interrupts and Permanent Address (MAC), verify using ifconfig command, something like this "Interrupt:64 Memory:fb000000-fb7fffff"

Way to check if the interface is physical - having interrupts in ifconfig output

$ ls -l /sys/class/net/
total 0
lrwxrwxrwx 1 root root    0 Jun 23 16:49 bond0 -> ../../devices/virtual/net/bond0
lrwxrwxrwx 1 root root    0 Jun 23 16:49 bond0.1 -> ../../devices/virtual/net/bond0.1
lrwxrwxrwx 1 root root    0 Jun 23 16:49 bond1 -> ../../devices/virtual/net/bond1
lrwxrwxrwx 1 root root    0 Jun 23 16:49 bond1.2 -> ../../devices/virtual/net/bond1.2
lrwxrwxrwx 1 root root    0 Jun 23 16:49 eth01 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.0/net/eth01
lrwxrwxrwx 1 root root    0 Jun 23 16:49 eth02 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.1/net/eth02
lrwxrwxrwx 1 root root    0 Jun 23 16:49 lo -> ../../devices/virtual/net/lo

$ sudo find / -name operstate

/sys/devices/virtual/net/lo/operstate

/sys/devices/pciXYZ/net/eth01/operstate

/sys/devices/pciXYZ/net/eth02/operstate

cat /sys/devices/pciXYZ/net/eth02/operstate

up

$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth01
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Slave Interface: eth01
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 2c:59:e5:3f:28:48
Slave queue ID: 0
Slave Interface: eth02
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 2c:59:e5:3f:28:4c
Slave queue ID: 0

Parameters explaination -
Bonding Mode: fault-tolerance (active-backup) - only one (slave) will be active at a time. only when first one fails, second becomes active. Upon master interface failure, bonding will issue Gratitous ARP, One gratutious ARP is issued for the bonding master interface and each VLAN interfaces configured above it,
other options are
=> balance-rr or 0 
=> active-backup or 1 
=> balance-xor or 2 
=> broadcast or 3
=> 802.3ad or 4
=> balance-tlb or 5
=> balance-alb or 6 
Primary Slave: None
Currently Active Slave: eth01 - specifying which slave is the primary device. Only applicable for Active-backup mode.
It indicates the preferred interface to be Active when Available.
MII Status: up - Shows if the bond status is Active/ Up
MII Polling Interval (ms): 600 - Duration for polling when the Active interface goes down
Bonding needs a method to monitor its slave link, otherwise it doesn't know the link state of the slave, nor does bonding react when the slave link state changes. The two monitors are the miimon which watches physical link, and the ARP monitor which uses traffic but isn't compatible with LACP.
The default should be to miimon at an interval of 100ms. It's not clear why this didn't apply to your systems. It's a good idea to always specify the proper link monitoring mode.
Up Delay (ms): 200 - Specifies (in milliseconds) how long to wait before enabling a link. The value must be a multiple of the value specified in the miimon parameter. The value is set to 0 by default, which disables it.
Down Delay (ms): 200 - How long to wait after link failure before disabling the link. 
Link Failure Count - This shows up for times the link went down/up
reference -  https://www.kernel.org/doc/Documentation/networking/bonding.txt

additional settings
arp_interval=time_in_milliseconds
Specifies, in milliseconds, how often ARP monitoring occurs.
The value is set to 0 by default, which disables it.
arp_ip_target=ip_address[,ip_address_2,…ip_address_16]
Specifies the target IP address of ARP requests when the arp_interval parameter is enabled. Up to 16 IP addresses can be specified in a comma separated list.
arp_validate=value
Validate source/distribution of ARP probes; default is none. Other valid values are active, backup, and all.
reference - http://linux-ip.net/html/ether-bonding.html

[root@real-server root]# modprobe bonding mode=1 miimon=100 downdelay=200 updelay=200
[root@real-server root]# ip link set dev bond0 addr 00:80:c8:e7:ab:5c
[root@real-server root]# ip addr add 192.168.100.33/24 brd + dev bond0
[root@real-server root]# ip link set dev bond0 up
[root@real-server root]# ifenslave  bond0 eth2 eth3
The interface eth2 is up, shutting it down it to enslave it.
The interface eth3 is up, shutting it down it to enslave it.
[root@real-server root]# ip link show eth2 ; ip link show eth3 ; ip link show bond0
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 100
  link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,NOARP,SLAVE,DEBUG,AUTOMEDIA,PORTSEL,NOTRAILERS,UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 100
  link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff
58: bond0: <BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue
  link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff
https://blog.dbi-services.com/the-same-mac-address-for-two-interfaces-on-the-same-host/
When in high availability configuration, the MAC addresses will be same for all elements (bond, slave interfaces)
like here bond0 consists of enp0s9 and enp0s10 interfaces, so effectivly no impact of one interface failure
4: enp0s9:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
5: enp0s10:  mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
6: bond0:  mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:85:02:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.223/24 brd 192.168.22.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe85:2d8/64 scope link tentative 
       valid_lft forever preferred_lft forever

Note - to achieve different MAC addresses, set this flag fail_over_mac (more at https://www.kernel.org/doc/Documentation/networking/bonding.txt)
 sed -i 's/BONDING_OPTS="mode=active-backup miimon=100"/BONDING_OPTS="mode=active-backup miimon=100 fail_over_mac=active"/g' ifcfg-bond0 

by default this flag will be set to None
But assigning different MAC address has a sideeffect, wherein when ever interface change/ active goes  down the MAC address of the bond interface will change.

LACP - Link Aggregation Control Protocol  - https://en.wikipedia.org/wiki/Link_aggregation