{"id":184,"date":"2015-02-09T15:59:03","date_gmt":"2015-02-09T04:59:03","guid":{"rendered":"https:\/\/icicimov.com\/blog\/?p=184"},"modified":"2017-01-02T16:00:00","modified_gmt":"2017-01-02T05:00:00","slug":"sriov-enhanced-networking-in-aws-ec2-on-ubuntu-14-04-hvm","status":"publish","type":"post","link":"https:\/\/icicimov.com\/blog\/?p=184","title":{"rendered":"SRIOV Enhanced Networking in AWS EC2 on Ubuntu-14.04 HVM"},"content":{"rendered":"

The latest EC2 generation of HVM instances makes use of the Enhanced Networking, utilizing the ixgbevf e1000 Gigabit Virtual Function Network Driver<\/code> which provides significantly faster network layer processing. We can see it is already in use on Ubuntu-14.04<\/code> with kernel 3.0.43:<\/p>\n

$ modinfo ixgbevf\nfilename:       \/lib\/modules\/3.13.0-45-generic\/kernel\/drivers\/net\/ethernet\/intel\/ixgbevf\/ixgbevf.ko\nversion:        2.11.3-k\nlicense:        GPL\ndescription:    Intel(R) 82599 Virtual Function Driver\nauthor:         Intel Corporation, <linux.nics@intel.com>\nsrcversion:     AE2D8A25951B508611E943D\nalias:          pci:v00008086d00001515sv*sd*bc*sc*i*\nalias:          pci:v00008086d000010EDsv*sd*bc*sc*i*\ndepends:       \nintree:         Y\nvermagic:       3.13.0-45-generic SMP mod_unload modversions\nsigner:         Magrathea: Glacier signing key\nsig_key:        34:99:21:39:F3:DA:40:B6:20:BD:55:17:59:7B:A8:5A:F5:79:7C:9A\nsig_hashalgo:   sha512\nparm:           debug:Debug level (0=none,...,16=all) (int)\n\n$ sudo ethtool -i eth0\ndriver: ixgbevf\nversion: 2.11.3-k\nfirmware-version:\nbus-info: 0000:00:03.0\nsupports-statistics: yes\nsupports-test: yes\nsupports-eeprom-access: no\nsupports-register-dump: yes\nsupports-priv-flags: no\n<\/code><\/pre>\n
We need to upgrade to at least version 2.14.2 as recommended on the AWS web site since it contains lots of important bug fixes. First we upgrade to the latest kernel (3.0.45 atm)  and reboot:<\/p>\n
$ sudo aptitude update && sudo aptitude safe-upgrade -y\n$ sudo reboot\n<\/code><\/pre>\n
Then we download the latest version of the ixgbevf module, compile and install via DKMS:<\/p>\n
$ sudo aptitude install -y dkms\n$ wget http:\/\/sourceforge.net\/projects\/e1000\/files\/ixgbevf%20stable\/2.16.1\/ixgbevf-2.16.1.tar.gz\n$ tar -xzf ixgbevf-2.16.1.tar.gz\n$ sudo mv ixgbevf-2.16.1 \/usr\/src\/\n<\/code><\/pre>\n
Now, the compile on Ubuntu-14.04 fails since the distro is passing wrong kernel version to the compiler. There was a long awaited patch released on February 3th fixing this at Ubuntu-14.04.1 ixgbevf patch<\/a>. We backup the header file and apply the patch:<\/p>\n
$ sudo cp \/usr\/src\/ixgbevf-2.16.1\/src\/kcompat.h \/usr\/src\/ixgbevf-2.16.1\/src\/kcompat.h.orig\n<\/code><\/pre>\n
and after patching:<\/p>\n
$ sudo diff -u \/usr\/src\/ixgbevf-2.16.1\/src\/kcompat.h.orig \/usr\/src\/ixgbevf-2.16.1\/src\/kcompat.h\n--- \/usr\/src\/ixgbevf-2.16.1\/src\/kcompat.h.orig    2015-03-03 01:18:53.419899459 +0000\n+++ \/usr\/src\/ixgbevf-2.16.1\/src\/kcompat.h    2015-03-03 01:22:32.979899459 +0000\n@@ -3219,8 +3219,6 @@\n #define u64_stats_update_begin(a) do { } while(0)\n #define u64_stats_update_end(a) do { } while(0)\n #define u64_stats_fetch_begin(a) do { } while(0)\n-#define u64_stats_fetch_retry_bh(a) (0)\n-#define u64_stats_fetch_begin_bh(a) (0)\n\n #if (RHEL_RELEASE_CODE && RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(6,1))\n #define HAVE_8021P_SUPPORT\n@@ -4174,8 +4172,8 @@\n\n \/*****************************************************************************\/\n #if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,15,0) )\n-#define u64_stats_fetch_begin_irq u64_stats_fetch_begin_bh\n-#define u64_stats_fetch_retry_irq u64_stats_fetch_retry_bh\n+#define u64_stats_fetch_begin_irq(a) (0)\n+#define u64_stats_fetch_retry_irq(a, b) (0)\n #else\n #define HAVE_PTP_1588_CLOCK_PINS\n #define HAVE_NETDEV_PORT\n<\/code><\/pre>\n
Now we can proceed with the build. We create DKMS driver config file \/usr\/src\/ixgbevf-2.16.1\/dkms.conf<\/code> for the new module:<\/p>\n
PACKAGE_NAME=\"ixgbevf\"\nPACKAGE_VERSION=\"2.16.1\"\nCLEAN=\"cd src\/; make clean\"\nMAKE=\"cd src\/; make BUILD_KERNEL=${kernelver}\"\nBUILT_MODULE_LOCATION[0]=\"src\/\"\nBUILT_MODULE_NAME[0]=\"ixgbevf\"\nDEST_MODULE_LOCATION[0]=\"\/updates\"\nDEST_MODULE_NAME[0]=\"ixgbevf\"\nAUTOINSTALL=\"yes\"\n<\/code><\/pre>\n
then we build and install:<\/p>\n
$ sudo dkms build -m ixgbevf -v 2.16.1\n$ sudo dkms install -m ixgbevf -v 2.16.1\n$ sudo update-initramfs -c -k all\n<\/code><\/pre>\n
Now if we check the module version in use:<\/p>\n
$ modinfo ixgbevf\nfilename:       \/lib\/modules\/3.13.0-46-generic\/updates\/dkms\/ixgbevf.ko\nversion:        2.16.1\nlicense:        GPL\ndescription:    Intel(R) 10 Gigabit Virtual Function Network Driver\nauthor:         Intel Corporation, <linux.nics@intel.com>\nsrcversion:     3F8AACF779F38FD444B1CD3\nalias:          pci:v00008086d00001515sv*sd*bc*sc*i*\nalias:          pci:v00008086d000010EDsv*sd*bc*sc*i*\ndepends:       \nvermagic:       3.13.0-46-generic SMP mod_unload modversions\nparm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281, 0=off, 1=dynamic), default 1 (array of int)\n<\/code><\/pre>\n
Finally we need to reboot once more to put the new module in use:<\/p>\n
$ sudo shutdown -r now\n\n$ sudo ethtool -i eth0\ndriver: ixgbevf\nversion: 2.16.1\nfirmware-version: N\/A\nbus-info: 0000:00:03.0\nsupports-statistics: yes\nsupports-test: yes\nsupports-eeprom-access: no\nsupports-register-dump: yes\nsupports-priv-flags: no\n<\/code><\/pre>\n
The final step is enabling the instance SRIOV parameter so it can start utilizing the new driver:<\/p>\n
$ aws ec2 stop-instances --instance-ids instance_id\n$ aws ec2 modify-instance-attribute --instance-id instance_id --sriov-net-support simple\n$ aws ec2 start-instances --instance-ids instance_id\n<\/code><\/pre>\n
The HVM (this is not supported by PV instances) instances we use for our Joomla! setup had this already activated so it was not necessary in this case.<\/p>\n
For the end, although obviously providing huge benefit, this module needs to be rebuilt and reinstall upon every kernel upgrade.<\/p>\n","protected":false},"excerpt":{"rendered":"
The latest EC2 generation of HVM instances makes use of the Enhanced Networking, utilizing the ixgbevf e1000 Gigabit Virtual Function Network Driver which provides significantly faster network layer processing. We can see it is already in use on Ubuntu-14.04 with…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,10],"tags":[],"class_list":["post-184","post","type-post","status-publish","format-standard","hentry","category-aws","category-server"],"_links":{"self":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=184"}],"version-history":[{"count":1,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/184\/revisions"}],"predecessor-version":[{"id":185,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/184\/revisions\/185"}],"wp:attachment":[{"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icicimov.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}