2017/12/28

Run ARM executables on Linux x86 with Intel Houdini

This is only an experimental test. Please DO NOT abuse. Take responsibility for your own if using it in other purposes.

Intel Houdini is applied in x86 arch Android devices to enable the possibility of ARM code support. It's awesome and how about extending the possibility to Linux?

The Intel Houdini binaries is proprietary, you can only find them in several x86 model Android devices' vendor binaries. Nexus Player (fugu) contains currently the latest version that I can find.
https://developers.google.com/android/drivers#fuguopr2.170623.027
SHA256: 114c20b8335f3c166c76d590c6d238153c463c2fa2dea3da56d1ce3ffd1ec6ed

Accept the license at your own risk. Extract it and you get the Houdini and other unrelated binaries in vendor/intel/fugu/proprietary.
In vendor/intel/fugu/proprietary/device-partial.mk, files manifest is located here. All you need to do is copy them to your Linux host. However, the target path is Android style and this path style is hard-coded in Houdini binaries, so just take it easy, mkdir /system, try script below (root needed).

import os
import shutil

with open('device-partial.mk') as f:
    for line in f:
        line = line.strip()
        if line.startswith(r"vendor/intel/fugu/"):
                j = line.split(':')
                if len(j) < 2:
                    continue
                src = j[0]
                src = src[src.find(r"proprietary/"):]
                dst = '/'+j[1]
                dst = os.path.dirname(dst)
                if not os.path.exists(dst):
                    os.makedirs(dst)
                shutil.copy(src, dst)
                print("src %s copied to dst %s" % (src, dst))

Houdini relies binfmt_misc feature by Linux Kernel to pass the ARM executables to Houdini translator. You need to register ARM executable magic to Houdini translator /system/bin/houdini.

su
modprobe binfmt_misc
mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
cd vendor/intel/fugu/proprietary
cat arm_dyn > /proc/sys/fs/binfmt_misc/register
cat arm_exe > /proc/sys/fs/binfmt_misc/register

Next step, install ARM runtime libraries on Linux. For Ubuntu 16.04:

# For default APT servers on Ubuntu x86 provide no ARM support, you need to add restriction to fetch only i386, amd64 packages from main server. Example: deb [arch=i386,amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted
# Then add APT servers providing ARM packages.
deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ xenial main
deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main
deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ xenial-security main

dpkg --add-architecture armhf
apt update
apt install libc6:armhf

Now we need Houdini using Linux GNU linker instead of Android linker. And chmod.

su
cp /lib/arm-linux-gnueabihf/ld-2.23.so /system/lib/arm/linker
chmod 755 /system/lib/arm/linker
chmod 755 /system/bin/houdini

Almost done. Let's try run OpenSSL for benchmark. Firstly, download a ARMHF openssl:

apt install libssl1.0.0:armhf
# DO NOT apt install openssl:armhf for breaking many things.
# Extract openssl from http://ports.ubuntu.com/ubuntu-ports/pool/main/o/openssl/openssl_1.0.2g-1ubuntu4.10_armhf.deb

Here is the result:

# NATIVE
~/poc$ openssl speed -evp AES256
Doing aes-256-cbc for 3s on 16 size blocks: 90227643 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 24112794 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 6133873 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 1526964 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 192347 aes-256-cbc's in 3.00s
OpenSSL 1.0.2g  1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: cc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     481214.10k   514406.27k   523423.83k   521203.71k   525235.54k

# ARMHF
~/poc$ ./openssl speed -evp AES256
ERROR: ld.so: object '/system/lib/arm/libaeabi_map.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Doing aes-256-cbc for 3s on 16 size blocks: 10090430 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 3350791 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 878897 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 176189 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 27901 aes-256-cbc's in 3.00s
OpenSSL 1.0.2g  1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) 
compiler: cc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      53815.63k    71483.54k    74999.21k    60139.18k    76188.33k

Additional, to fix  '/system/lib/arm/libaeabi_map.so' from LD_PRELOAD warning.
The cause is that Houdini will set LD_PRELOAD=/system/lib/arm/libaeabi_map.so, libaeabi_map.so is not neccesary for us as it's used for patching libc.so and libm.so on Android.
So, NUL the path can silent the warning.
su
sed -i 's/\/system\/lib\/arm\/libaeabi_map\.so/\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0\x0/g' /system/bin/houdini

Bugs may exist for this Houdini binaries was not intended for Linux.

Google Pixel modem.img mod for Chinese carriers

Carriers in China like China Telecom was stuck on CSFB mode of 4G NOT globally popular VOLTE mode, so most cell-phones not selling in China ...