Sunday, March 24, 2013

The Evil Came Back: Darkleech's Apache Malware Module: Recent Infection, Reversing, Prevention & Source Details

With the help of Malware Researchers, & solid coordination with authorities and admins involved we successfully stopped the mass attack of current threat which damaged hundreds of Linux Apache web servers within 2 weeks infection periods. I thank you for authority who finally approved MalwareMustDie released the know how for this threat by this simple post. Credit list is written under this post.

Malware Definition and Historical Research

This definition is written by the latest observation of the large infection case occurred caused by this malware's infection on Linux Apache Web Servers.

Darkleech Apache Malware Module version (there is also detected an NGNIX version of this malware as per mentioned --> here by Eric Romang) is the malware implemented in Linux OS served with Apache web server with the Apache API interactive module. Malware module was loaded and activated into the Apache web server system by LoadModule command defined in the module configuration file. Once it is loaded into the system it perform general malicious functionality of: [1] Self injection of compromised server's web pages with the code to redirect victim to the malware sites and [2] Backdoor the compromised server system from the remote access.

Before starting the details of infection it is good to understand the background of malware to be about to discussed in this post:

This malware is already recognized by antivirus products with the reference name of Linux/Chapro.x or other names, you can seek it in Google--> here. Historically, this malware's infection attack was firstly exposed by Unmask Parasites on August 13th, 2012 in--> here and was firstly very detailed disclosed in October 2012 by russian malware researcher in PDF presentation shared at yandex.ru--> here. (Direct download PDF is--> here). And I believe the first english coverage of this malware was written by Unmask Parasites in--> here. Following by various anti virus research reports and coverage i.e. by ESET, Securelist, Symantec, etc.

The bad actor group who is using Darkleech Apache Module is the same bad guys behind the Cryptome infection more than 1 year ago, and LA Times incident a while ago. This was exposed in Malware Don't Drink Coffee blog in--> here. And the latest mass-infection detected using this malware was pointing to Blackhole Exploit Kit with spreading combination of Trojan PWS/Downloader collaborated with FakeAV and/or ZeroAccess malware, exposed in--> here.

The first time the related malware sample was uploaded to Virus Total is in--> here. And the link between Darkleech underground forum to this malware firstly exposed via Eric Romang's post in--> here, which pointing to the Russian underground forum (forum snapshot is below).

Latest Infection Details

The malware was found in web server systems with below characteristic:
Linux RedHat-base distribution without SE Linux properly set
Apache httpd web server 2.x (rpm-base, as per it is)
Cgi-base web admin panel and/or Wordpress system's served
Malware module file was found with the below regex:
mod\_[a-z0-9]{3,}\_[a-z0-9]{3,}\.so
With the below file names:
mod_sec2_config.so
mod_pool_log.so
mod_chart_proxy.so
mod_balance_alias.so
mod_local_log.so
mod_build_cache.so
  :
Malware was loaded in various malicious conf file with using Apache module's LoadModule method below:
$ cat ../etc/../modules/[VARIOUS].conf| grep "mod_"
LoadModule sec2_config_module modules/mod_sec2_config.so
PS: malware module files was using old dates.

Infection Symptoms

When an Apache web server get infected by this malware it shows the unwanted redirection to the remote web servers served with the malware infection codes, mostly are Exploit Kit's landing page. One real infection session is shown as below PCAP record:


Landing page like:

Before redirection occurred the malware injection code will 
be detected in the previous HTTP GET traffic in the 
infected server as per below PCAP (see the 1st request)

In the first traffic we'll see the malware injected code:

contains the javascript wrapped iframe code like below:

which will trigger the malware downloads like real sample below:


Infection Condition

There are several malware infection conditions that "supposed" to be met for an infection. All of the prerequisites for infection was defined in the client's HTTP access to the infected web servers, also in the checked values of the infected web servers. Unmask Parasites blog in--> here is explaining the very useful guides for us to break down the recent malware Apache module spotted, below are the details:
 
1. The usage of referer to block unwanted browsers.
By reversing, we found the malware has function 
(C_ARRAY_BAN_USERAGENT) to ban unwanted browsers 
with the below list:
SAFARI             YANDEX
OPERA              CRAWLER
FIREFOX            JIKE
CHROME             SPIDER
GOOGLEBOT          ROBOT
SLURP              PAPERLIBOT
YAHOO              SNAPPREVIEWBOT
BING               BUFFERBOT
LINUX              MEDIAPARTNERS
OPENBSD            HATENA
MACINTOSH          BLUEDRAGON
MAC OS             WORDPRESS
IPHONE             XIANGUO
SYMBIANOS          WOOPINGBOT
NOKIA              CAFFEINATED
LINKDEX            FEEDZIRRA
FROG/1             BITLYBOT
USER-AGENT         FOIIABOT
BLACKBERRY         PROXIMIC
MOTOROLA           VBSEO
APPLE-PUB          FOLLOWSITE
AKREGATOR          SOGOU
SONYERICSSON       NHN
MACBOOK            WGET
XENU LINK          MSNBOT
METAURI            YOUDAO
REEDER             STACKRAMBLER
MOODLEBOT          LWP::SIMPLE
SAMSUNG            QIHOOBOT
SINDICE-FETCHER    BRUTUS
EZOOMS             HTTPCLIENT
NIKOBOT            NIELSEN
BINLAR             CURL
DARWIN             PHP
PLAYSTATION        INDY LIBRARY
OPERA MINI         NINTENDO
2. Checked referer sites Detected malware modules was checking below sites before injecting the redirection code, spotted in value of C_ARRAY_SE_REFERRER below:
GOOGLE.     ICQ.   
YAHOO.      NETZERO.  
YANDEX.     FRESH-WEATHER.   
RAMBLER.    FREECAUSE.    
MAIL.RU     MYSEARCH-FINDER.   
BING.       NEXPLORE. 
SEARCH.     ATT.   
MSN.        REDROVIN.
ALLTHEWEB.  TOSEEKA.      
ASK.        COMCAST.
LOOKSMART.  INCREDIMAIL.      
ALTAVISTA.  CHARTER.      
WEB.DE      VERIZON.  
FIREBALL.   SUCHE.     
LYCOS.      VIRGILIO.  
AOL.        VERDEN.
After some tests on the infected sites we found that the referer below was not infecting the sites. 3. Others malware blacklist method:
The malware tries to identify unwanted access from web site admins, server login admins, server unwanted processes as per spotted malicious functions below:
C_ARRAY_BAN_LOCAL_IP
   (contains IP addresses)
C_ARRAY_BLACKLIST_URI
     "ADMIN"
C_ARRAY_SUDOERS
   (contains list of user with sudoers right)
C_ARRAY_BAN_PROC
   (contains MD5 of banned process)
   i.e.:
   f7277f6714e4b034216cf6558cc6327b
   28878074a3dd19c7361e8a6d3f04fc17
   d0415afe195478d4d8c9af205644
4. Malware checked conditions:
The malware is having below condition to be passed for performing infection as per described in the below details:
_CHECK_BLACKLIST           0x3D20  
_CHECK_BOT_USERAGENT       0x3650  
_CHECK_JS                  0x3180  
_CHECK_LOCAL_IP            0x44F0  
_CHECK_PROC                0x3980  
_CHECK_RAW_COOKIE          0x3190  
_CHECK_REFERER_IS_HOST     0x31C0  
_CHECK_REFERER_IS_SEO      0x3540  
_CHECK_SITE_ADMIN          0x3860  
_CHECK_SITE_KERNEL         0x31B0  
_CHECK_UTMP                0x3BB0  
_CHECK_WAITLIST            0x5500  
5. The Blacklist files
We spotted the blacklist was saved in the temporary directory set in the Linux system environment TEMP or TMP variable (i.e.: "/var/tmp/" or also "/")under file names:
/var/tmp/sess_
Which is having various value of blacklist data described in the above conditions.

6. Usage of cookies to control infection
During investigation our friend in crusade @it4sec offered help on analysis infection condition. He found and posted the good theory of malware usage of cookies his blog "On Daily Basis" in--> here. Which I recommend you to read.

7. Finally, the post-check, code injection Injection methods:
INJECT
javascript
text/js
Injection Code Boundaries
C_MARKER_LEFT: {{{
C_MARKER_RIGHT: }}}
Inject commands:
_INJECT_DO
_INJECT_LOAD
_INJECT_SAVE
_INJECT_SKIP
_INJECT_UPDATE

Reversing Darkleech Malware Module

We are in coordination with jvoisin of dustri.org for reversing the current module under pure unix environment using python logic. jvoisin was making the best reversing method for this malware as per mentioned in his blog in--> here, which I recommend you to read. Also I used reference of an older version of this malware module source code spotted in pastebin in--> here.

Below is the reversing steps of the recent modules spotted between March 17th to March 22nd, 2013 in hundreds of infected sites we cleaned up. We used two permitted samples to be uploaded into virus total as per below details:

Sample 1 URL --> here
SHA256: 94ef407cc485989464dcf390fcea6e82218bc89f75394e41a95e0bb31830786b
SHA1: cc594b4d924b0710db64bcca5012d22db8842f98
MD5: 81c1d493c7764f6692c30de8923c76ba
File size: 36.4 KB ( 37296 bytes )
File name: mod_sec2_config.so
File type: ELF
Tags: elf
Detection ratio: 4 / 45
Analysis date: 2013-03-20 02:42:20 UTC ( 5 minutes ago )
【ExifTool】
MIMEType.................: application/octet-stream
CPUByteOrder.............: Little endian
CPUArchitecture..........: 32 bit
FileType.................: ELF executable
ObjectFileType...........: Shared object file
CPUType..................: i386
【Malware Name】
GData                    : ELF:Apmod-B
Avast                    : ELF:Apmod-B [Trj]
Microsoft                : Backdoor:Linux/Apmod.gen!A
Kaspersky                : HEUR:Backdoor.Linux.Apmod.gen
Sample 2 URL --> here
SHA256: ece16200fd54500a33d81f37a9f864148cbf8846514978413168ffacd46d28c3
SHA1: ef3741f3cc2c60cc4cd88e6293776e39d56cd78b
MD5: ae7c369b8bd49a04f87fab72d4d3431d
File size: 36.4 KB ( 37272 bytes )
File name: mod_pool_log.so
File type: ELF
Tags: elf
Detection ratio: 5 / 45
Analysis date: 2013-03-20 02:42:45 UTC ( 9 minutes ago )
【ExifTool】
MIMEType.................: application/octet-stream
CPUByteOrder.............: Little endian
CPUArchitecture..........: 32 bit
FileType.................: ELF executable
ObjectFileType...........: Shared object file
CPUType..................: i386
【Malware Name】
GData                    : ELF:Apmod-B
Avast                    : ELF:Apmod-B [Trj]
Microsoft                : Backdoor:Linux/Apmod.gen!A
Kaspersky                : HEUR:Backdoor.Linux.Apmod.gen
Ikarus                   : Backdoor.Linux.Apmod
File Information (the first file only)
$ ls -alF ./mod_sec2_config.so
-rwxr--r--  1 xxx xxx  37296 Jun 26  2007 ./mod_sec2_config.so*

// md5..

$ md5 mod_sec2_config.so
MD5 (mod_sec2_config.so) = 81c1d493c7764f6692c30de8923c76ba

// file info

file format elf32-i386
architecture: i386, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x00003050

0000   7F 45 4C 46 01 01 01 00 00 00 00 00 00 00 00 00    .ELF............
0010   03 00 03 00 01 00 00 00 50 30 00 00 34 00 00 00    ........P0..4...
0020   A0 8D 00 00 00 00 00 00 34 00 20 00 05 00 28 00    ........4. ...(.
0030   1A 00 19 00 01 00 00 00 00 00 00 00 00 00 00 00    ................
0040   00 00 00 00 F0 77 00 00 F0 77 00 00 05 00 00 00    .....w...w......
0050   00 10 00 00 01 00 00 00 00 80 00 00 00 80 00 00    ................
0060   00 80 00 00 F0 0B 00 00 84 0D 00 00 06 00 00 00    ................
0070   00 10 00 00 02 00 00 00 18 80 00 00 18 80 00 00    ................
0080   18 80 00 00 D0 00 00 00 D0 00 00 00 06 00 00 00    ................
0090   04 00 00 00 50 E5 74 64 C0 70 00 00 C0 70 00 00    ....P.td.p...p..
00A0   C0 70 00 00 7C 01 00 00 7C 01 00 00 04 00 00 00    .p..|...|.......
00B0   04 00 00 00 51 E5 74 64 00 00 00 00 00 00 00 00    ....Q.td........
  :
For unixmen, fire your objdump to gain these values:
Program Header:
    LOAD off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**12
         filesz 0x000077f0 memsz 0x000077f0 flags r-x
    LOAD off    0x00008000 vaddr 0x00008000 paddr 0x00008000 align 2**12
         filesz 0x00000bf0 memsz 0x00000d84 flags rw-
 DYNAMIC off    0x00008018 vaddr 0x00008018 paddr 0x00008018 align 2**2
         filesz 0x000000d0 memsz 0x000000d0 flags rw-
EH_FRAME off    0x000070c0 vaddr 0x000070c0 paddr 0x000070c0 align 2**2
         filesz 0x0000017c memsz 0x0000017c flags r--
   STACK off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**2
         filesz 0x00000000 memsz 0x00000000 flags rw-

Dynamic Section:
  NEEDED      libm.so.6
  NEEDED      libc.so.6
  SONAME      mod_sec2_config.so
  INIT        0x29f8
  FINI        0x6f74
  HASH        0xd4
  STRTAB      0x16ec
  SYMTAB      0x7bc
  STRSZ       0x964
  SYMENT      0x10
  PLTGOT      0x82cc
  PLTRELSZ    0x318
  PLTREL      0x11
  JMPREL      0x26e0
  REL         0x22c8
  RELSZ       0x418
  RELENT      0x8
  VERNEED     0x2238
  VERNEEDNUM  0x2
  VERSYM      0x2050
  RELCOUNT    0x9

Version References:
  required from libm.so.6:
    0x0d696910 0x00 08 GLIBC_2.0
  required from libc.so.6:
    0x09691f73 0x00 07 GLIBC_2.1.3
    0x0d696911 0x00 06 GLIBC_2.1
    0x0d696914 0x00 05 GLIBC_2.4
    0x09691974 0x00 04 GLIBC_2.3.4
    0x0d696913 0x00 03 GLIBC_2.3
    0x0d696910 0x00 02 GLIBC_2.0

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .hash         000006e8  000000d4  000000d4  000000d4  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .dynsym       00000f30  000007bc  000007bc  000007bc  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .dynstr       00000964  000016ec  000016ec  000016ec  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .gnu.version  000001e6  00002050  00002050  00002050  2**1
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .gnu.version_r 00000090  00002238  00002238  00002238  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .rel.dyn      00000418  000022c8  000022c8  000022c8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .rel.plt      00000318  000026e0  000026e0  000026e0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .init         00000017  000029f8  000029f8  000029f8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  8 .plt          00000640  00002a10  00002a10  00002a10  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  9 .text         00003f24  00003050  00003050  00003050  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 10 .fini         0000001c  00006f74  00006f74  00006f74  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 11 .rodata       0000011d  00006fa0  00006fa0  00006fa0  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 12 .eh_frame_hdr 0000017c  000070c0  000070c0  000070c0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 13 .eh_frame     000005b4  0000723c  0000723c  0000723c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 14 .ctors        00000008  00008000  00008000  00008000  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 15 .dtors        00000008  00008008  00008008  00008008  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 16 .jcr          00000004  00008010  00008010  00008010  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 17 .data.rel.ro  00000004  00008014  00008014  00008014  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 18 .dynamic      000000d0  00008018  00008018  00008018  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 19 .got          000001e4  000080e8  000080e8  000080e8  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 20 .got.plt      00000198  000082cc  000082cc  000082cc  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 21 .data         00000770  00008480  00008480  00008480  2**5
                  CONTENTS, ALLOC, LOAD, DATA
 22 .bss          00000184  00008c00  00008c00  00008bf0  2**5
                  ALLOC
 23 .comment      000000e4  00000000  00000000  00008bf0  2**0
                  CONTENTS, READONLY
Full strings used:
0x16ED   0x16ED   __gmon_start__
0x16FC   0x16FC   _init
0x1702   0x1702   _fini
0x1708   0x1708   __cxa_finalize
0x1717   0x1717   _Jv_RegisterClasses
0x172B   0x172B   to_hex
0x173A   0x173A   _CHECK_JS
0x1744   0x1744   _CHECK_RAW_COOKIE
0x1756   0x1756   KEY_CLIENT
0x1761   0x1761   _CHECK_SITE_KERNEL
0x1774   0x1774   _CHECK_REFERER_IS_HOST
  :        :        
0x8BF1   0x8BF1   GCC: (GNU) 4.1.1 20060525 (Red Hat 4.1.1-1)
0x8C1E   0x8C1E   GCC: (GNU) 4.1.1 20070105 (Red Hat 4.1.1-51)
0x8C4C   0x8C4C   GCC: (GNU) 4.1.1 20070105 (Red Hat 4.1.1-51)
0x8C7A   0x8C7A   GCC: (GNU) 4.1.1 20070105 (Red Hat 4.1.1-51)
0x8CA8   0x8CA8   GCC: (GNU) 4.1.1 20060525 (Red Hat 4.1.1-1)
  :         :
0x8D55   0x8D55   .ctors
0x8D5C   0x8D5C   .dtors
0x8D68   0x8D68   .data.rel.ro
0x8D75   0x8D75   .dynamic
0x8D83   0x8D83   .got.plt
0x8D8C   0x8D8C   .data
0x8D97   0x8D97   .comment
Here's the full list--> here Separating import modules & symbols values: Imported modules:
$ rabin2 -i ./mod_sec2_config.so | cut -d" " -f7 | cut -c6- | sort
[Imports]

67 imports
_Jv_RegisterClasses
__ctype_b_loc
__ctype_tolower_loc
__ctype_toupper_loc
__cxa_finalize
__fprintf_chk
__gmon_start__
__memcpy_chk
     :
Symbols...
$ rabin2 -s ./mod_sec2_config.so | cut -d" " -f8 | cut -c6- | sort
[Symbols]

163 symbols
ARRAY_BAN_LOCAL_IP
ARRAY_BAN_PROC
ARRAY_BAN_USERAGENT
ARRAY_BLACKLIST_URI
ARRAY_SE_REFERER
ARRAY_SUDOERS
  :
Full list of imported modules & symbols--> here We'll the XOR endoded strings like: At the last part of reversed symbols we found XOR functions:
0x17C8   0x17C8   xor_decrypt_string
0x17ED   0x17ED   xor_encrypt_string
0x1800   0x1800   xor_encrypt
I reversed them to confirm XOR method:
xor_encrypt(A8, Ac, A10, A14)
/* unknown */ void  A8;
/* unknown */ void  Ac;
/* unknown */ void  A10;
/* unknown */ void  A14;
{
 /* unknown */ void  ebx;
 /* unknown */ void  esi;
 /* unknown */ void  Vfffffff4;

    edx = A10;
    L00003117();
    ebx = ebx + 0x4f3b;
    if(edx != 0 && A8 != 0) {
        Vfffffff4 = A14;
        *esp = *( *( *( *(ebx + -300)) + 0xc));
        *(ebp - 0x10) = L00002D90();
        if(A14 > 0) {
            ecx = 0;
            do {
                edx = ecx;
                eax = ecx;
                edx = edx >> 0x1f;
                Ac = Ac / Ac;
                eax = *(ecx + A10) & 0xff;
                al = al ^ *(Ac % Ac + A8);
                *(ecx + *(ebp - 0x10)) = al;
                ecx = ecx + 1;
            } while(ecx != A14);
        }
        return *(ebp - 0x10);
    }
    *(ebp - 0x10) = 0;
    eax = *(ebp - 0x10);
    esp = esp + 0xc;}

xor_encrypt_string(A8, Ac, A10, A14)
/* unknown */ void  A8;
/* unknown */ void  Ac;
/* unknown */ void  A10;
/* unknown */ void  A14;
{
 /* unknown */ void  V0;
 /* unknown */ void  V4;
 /* unknown */ void  ebx;
 /* unknown */ void  Vfffffffc;

    ebx = ebx + 0x4f7d;
    V4 = L00003117();
    V0 = A10;
    Vfffffffc = Ac;
    *esp = A8;
    return L00002C00();}

xor_decrypt_string(A8, Ac, A10, A14)
/* unknown */ void  A8;
/* unknown */ void  Ac;
/* unknown */ void  A10;
/* unknown */ void  A14;
{
 /* unknown */ void  ebx;
 /* unknown */ void  esi;
 /* unknown */ void  Vfffffff4;

    L00003117();
    ebx = ebx + 0x5001;
    esp = esp - 0xc;
    Vfffffff4 = A14 + 1;
    *esp = *( *( *( *(ebx + -300)) + 0xc));
    *(ebp - 0x10) = L00002D90();
    if(A14 > 0) {
        ecx = 0;
        do {
            edx = 0;
            eax = 0;
            edx = 0 >> 0x1f;
            Ac = Ac / Ac;
            al = *A10 & 0xff ^ *(Ac % Ac + A8);
            *( *(ebp - 0x10)) = al;
        } while(1 != A14);
    }
    esi = *(ebp - 0x10);
    *(esi + A14) = 0;
    eax = esi;
    esp = esp + 0xc;}
So it looks decode and encode the XOR'ed strings. Question: what strings? It is actually contains malware hidden data:
C_MODULE_VERSION: 
C_CC_HOST: 
C_CC_URI: 
C_CC_REQUEST_FORMAT: 
C_MARKER_LEFT: 
C_MARKER_RIGHT:
C_TMP_DIR: 
C_LIST_PREF: 
C_COOKIE_NAME: 
C_ARRAY_TAGS_FOR_INJECT:
C_ARRAY_BAN_USERAGENT: 
C_ARRAY_BLACKLIST_URI:
C_ARRAY_SE_REFERRER: 
C_ARRAY_SUDOERS: 
C_ARRAY_BAN_PROC: 
C_ARRAY_BAN_LOCAL_IP:
C_STRING_1: 
C_STRING_2: 
C_STRING_3: 
   :
C_STRING_35:
C_STRING_36:
We need to find size & offset per variable, i.e.:
{'name':'C_MODULE_VERSION',       'size':10, 'offset':0x8491},
{'name':'C_CC_HOST',              'size':12, 'offset':0x849b},
{'name':'C_CC_URI',               'size':15, 'offset':0x84a7},
{'name':'C_CC_REQUEST_FORMAT',    'size':96, 'offset':0x84c0},
{'name':'C_MARKER_LEFT',          'size':3,  'offset':0x8520},
{'name':'C_MARKER_RIGHT',         'size':3,  'offset':0x8523},
{'name':'C_TMP_DIR',              'size':8,  'offset':0x8526},
{'name':'C_LIST_PREF',            'size':5,  'offset':0x852e},
{'name':'C_COOKIE_NAME',          'size':15, 'offset':0x8533},
{'name':'C_ARRAY_TAGS_FOR_INJECT','size':77, 'offset':0x8560},
{'name':'C_ARRAY_BAN_USERAGENT',  'size':622,'offset':0x85c0},
{'name':'C_ARRAY_BLACKLIST_URI',  'size':5,  'offset':0x882e},
{'name':'C_ARRAY_SE_REFERRER',    'size':281,'offset':0x8840},
{'name':'C_ARRAY_SUDOERS',        'size':1,  'offset':0x8959},
{'name':'C_ARRAY_BAN_PROC',       'size':94, 'offset':0x8960},
{'name':'C_ARRAY_BAN_LOCAL_IP',   'size':48, 'offset':0x89e0},
{'name':'C_STRING_1',             'size':12, 'offset':0x8a10},
{'name':'C_STRING_2',             'size':9,  'offset':0x8a1c},
{'name':'C_STRING_3',             'size':1,  'offset':0x8a25},
    :         :                      :
{'name':'C_STRING_33',            'size':20, 'offset':0x8b3e},
{'name':'C_STRING_34',            'size':1,  'offset':0x8b52},
{'name':'C_STRING_35',            'size':3,  'offset':0x8b5a}
And figured the XOR offset & size, i.e. at the second sample:
0x0000847a (02) 0000                   ADD [EAX], AL
0x0000847c (02) 0000                   ADD [EAX], AL
0x0000847e (02) 0000                   ADD [EAX], AL
0x00008480 (06) dc9ba14f377b           FCOMP QWORD [EBX+0x7b374fa1] // <==
0x00008486 (01) 40                     INC EAX
0x00008487 (04) c114ca42               RCL DWORD [EDX+ECX*8], 0x42
0x0000848b (02) ff08                   DEC DWORD [EAX]
0x0000848d (01) 16                     PUSH SS
0x0000848e (01) 95                     XCHG EBP, EAX
0x0000848f (05) 3544eeab90             XOR EAX, 0x90abee44
0x00008494 (02) 7d19                   JGE 0x000084af   ; 1
Then put the offset & size in the jvoisin's script:
// first sample:
fd.seek(0x8480)
key = fd.read(17)

//second sample:
fd.seek(0x84a0)
key = fd.read(23)
And we got the XOR decoded output values like:
$ python sec2.py "./mod_sec2_config.so"
C_MODULE_VERSION: "2012.12.14"
C_CC_HOST: "217.23.13.6"
C_CC_URI: "/Home/index.php"
C_CC_REQUEST_FORMAT: "POST %s HTTP/1.1"
Host: "%s"
Content-Type: "application/x-www-form-urlencoded"
Content-Length:"
    %i
    %s"
C_MARKER_LEFT: "{{{"
C_MARKER_RIGHT: "}}}"
C_TMP_DIR: "/"
C_LIST_PREF: "sess_"
C_COOKIE_NAME: "PHP_SESSION_ID="
C_ARRAY_TAGS_FOR_INJECT: "
    </script>
    </style>
    </head>
    </title>
    </body>
    </html>
    </table>
    </h1>
    </i>
    </ul>"
     :
or
$ python sec3.py "./mod_pool_log.so"
C_MODULE_VERSION: "2012.12.14"
C_CC_HOST: "217.23.13.65"
C_CC_URI: "/Home/index.php"
C_CC_REQUEST_FORMAT: "POST %s HTTP/1.1"
Host: "%s"
Content-Type: "application/x-www-form-urlencoded"
Content-Length: 
  "%i"
  "%s"
C_MARKER_LEFT: "{{{"
C_MARKER_RIGHT: "}}}"
C_TMP_DIR: "/var/tmp"
C_LIST_PREF: "sess_"
C_COOKIE_NAME: "PHP_SESSION_ID="
   :
You'll see the injection method used:
C_STRING_2: text/html
C_STRING_3: %
C_STRING_5: document.write('%s');
C_STRING_5: r
C_STRING_6: User-Agent
C_STRING_7: %s%.*s
C_STRING_8: Referer
C_STRING_9: X-Forwarded-For
C_STRING_10: Client-IP
C_STRING_11: X-Real-IP
C_STRING_12: Cookie
C_STRING_13: ;
C_STRING_14: %s/%s%s
C_STRING_15: INJECT
C_STRING_16: javascript
C_STRING_17: text/js
C_STRING_18: j
Injected code is saved in %s as as per PoC'ed in traffic PCAP: These are ment to inject redirection code after tags:
C_ARRAY_TAGS_FOR_INJECT: 
   </script>
   </style>
   </head>
   </title>
   </body>
   </html>
   </table>
   </h1>
   </i>
   </ul>
Like the below code (usage after tag): The ID, Cookies and Hashes used:
C_COOKIE_NAME: PHP_SESSION_ID=
C_STRING_20: id=
C_STRING_21: %a %d-%b-%Y %H:%M:%S %Z
C_STRING_22: Set-Cookie
C_STRING_23: %s%i; expires=%s; path=/
C_STRING_24: Set-Cookie
C_STRING_25: w
C_STRING_26: %
C_STRING_27: Request-Hash
Contacting mother ships with method:
C_CC_HOST: 217.23.13.65
C_CC_URI: /Home/index.php
C_CC_REQUEST_FORMAT: POST %s HTTP/1.1
Host: %s
Content-Type: application/x-www-form-urlencoded
Content-Length: %i

Infection method & traces

We positively PoC'ed web admins (i.e.: cPanel, Plesk, Webmin and WordPress) panel vulnerability was used as main vector of penetration into vectim's machine in this attack. In the current incident there are mostly same web admins panel (Plesk) were detected in almost all infected servers as per this list, with the same vulnerability. It looks like the attackers were beforehand well-prepared with some penetration method to gain web exploitation which were used to gain shell access and did the privilege escalation unto root. (I am not allowed to expose this detail further at this moment). Since the root were gained in all infected servers, there is no way we can trust the host or its credentials anymore, we suggest you to offline the machine and use the backup data to start new service, AND remember to change all of server's user since there are strong possibilities the leaked server's admin credentials.

One should not try to seek traces on infection in /var/log/messages, is useless since the related logs were deleted. But I advise to go straight to see the traces in the Apache modules directories to grep the rogue module filenames with the above described regex, or see the TMP or TEMP environment for the "sess_" malware's blacklist / saved files.


Attack source IP

These are redirected IPs & source of preliminary attack:
65.75.139.229
129.121.99.242
129.121.176.15
149.47.146.13
149.47.146.139
173.192.50.193
  :
These USA networks were used as EK redirection:
65.75.190.0/18,19,20,24 https://twitter.com/MalwareMustDie/status/313007473546117120
69.50.224.0/19 https://twitter.com/MalwareMustDie/status/313002510199693312
69.89.0.0/20 https://twitter.com/MalwareMustDie/status/312999183130968064
129.121.0.0/16, https://twitter.com/MalwareMustDie/status/312995306113466368
149.47.0.0/16 https://twitter.com/MalwareMustDie/status/312991655429033985 
And latest attack source detected was moved to Netherland:
217.23.13.65

Additional

The malware is not 100% working in some infected systems. In some systems it crashes with signal 11:
execve("./mod_sec2_config.so",
["./mod_sec2_config.so"],
[/* 21 vars */]) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Blocking below libs from unknown usage willl help:
/usr/lib/libbdl.so.0
/usr/lib/libm.so.6
/usr/lib/libc.so.6
Blocking unknown user access to this path will help:
/proc
/proc/%s/comm
/var/run/utmp
GLIBC version used to run malware module (to block):
    0x09691f73 0x00 07 GLIBC_2.1.3
    0x0d696911 0x00 06 GLIBC_2.1
    0x0d696914 0x00 05 GLIBC_2.4
    0x09691974 0x00 04 GLIBC_2.3.4
    0x0d696913 0x00 03 GLIBC_2.3
    0x0d696910 0x00 02 GLIBC_2.0
Encoding used:
base64decode
base64encode
to_hex
urlencode
xor_decrypt_string
xor_encrypt
xor_encrypt_string
Reversing Notes: Disassembly of malware functions is--> here. Complete disassembly of malware module is--> here.

Samples

For raising the AV's detection ratio and research, we are sharing the samples --> here.

Credits

Thank you to the wonderful individuals who help us in detection, analysis, cooperation for current threat handling: @kafeine - without him infection will be wide-spread. Denis Sinegubko of Unmask Parasites who wrote good facts of the threat from beginning! jvoisin - without this genious young man I will stuck in reversing the XOR'ed string Denis Laskov - good analysis for malware module's cookie scheme for infection Jim Kesselring - the MMD "Razor" to shut all US based infection Eric Romang - your related report helped a lot, specially NGNIX part, you know that? :-) David Harley - for the clarification of Linux/Chapro facts To all MalwareMustDie members & supporters involved in this investigation, This post is dedicated to sleepless hundreds admins who did great job in removing malwares, reinstalling and re-tuning their website due to this incident.
#MalwareMustDie!

15 comments:

  1. Hello.

    Thank you for reporting on this awful thing.

    I am a stoopid noob.
    How can I check my own system for infection?

    ReplyDelete
  2. There is no easy way, sir. And that will be depending on stage of infection itself.
    If you found your web site redirecting a unwanted request to remote site via :
    - any unwanted HTTP/1.1 POST request
    - any GET request which you are sure that is not related to your site's code.
    It means you are infected by this malware and the below method to search malware files can be found:

    1. Search in the Apache module configuration the usage of unwanted module file, like mentioned above post. Malware module file was found with the below regex (for search string purpose):
    mod\_[a-z0-9]{3,}\_[a-z0-9]{3,}\.so

    Malware module names spotted with the below similar file names:
    mod_sec2_config.so
    mod_pool_log.so
    mod_chart_proxy.so
    mod_balance_alias.so
    mod_local_log.so
    mod_build_cache.so
    :

    Malware was loaded in various malicious conf file with
    using Apache module's LoadModule method below:
    $ cat ../etc/../modules/[VARIOUS].conf| grep "mod_"
    LoadModule sec2_config_module modules/mod_sec2_config.so

    PS: malware module files was using old dates, so not waste time to search sorted by timestamp.

    2. Search in temporary directory set in the Linux system environment TEMP or TMP variable (i.e.: "/var/tmp/" or "/tmp" OR also can be "/") under file names: "%UNIX-TMP%/sess_*"

    If you found these symptoms better change all ssh unix user & password up to root, re-install the apache with the new latest version and restore your web site backup. PS: If I were you I cannot trust the server anymore and will start the backup one or the new new clean one with restoring all services/backup.
    You don't know what bad guys did after they gained the root access...

    ReplyDelete
  3. What's the decompress password for the sample?

    Thanks

    ReplyDelete
    Replies
    1. Mention me (@malwaremustdie) via twitter using your own account and we explain you the email address where you can get the password. I know it sounds too complicated, we just try prevent the sample goes to the wrong hand untraceable. Thank's for your understanding.

      Delete
    2. Hi,

      We also received a warning that some of our servers were infected. In order to verify this, I wrote a small script, but I couldn't test it yet. May I also get the password?

      Thanks in advance
      Stef

      Delete
    3. Poke us in twitter on @malwaremustdie and I will DM you the information.

      Delete
  4. Thank you guys a thousand times for investigation, exposing and explaining these quite complicated details! Very appreciated!

    ReplyDelete
    Replies
    1. This is a work of a team, friend.
      Let's thank to all people who sacrifice their free/rest time for analyzing hard stuff like this.

      Delete
  5. Since so many questions about how the attacker gain root privilege in this attack,
    Below I add the information of how the infected servers got hacked:

    In our case, it was postive confirmed the “CVE-2012-1557″Plesk vulnerability was used to gain control of shell to gain root privilege escalation in servers attacked in post infection.
    PoC of root escalation taken was /var/log/messages & several root permission directory was bypasse by the attacker. yes this is for sure.

    FYI, the details of the CVE used was: Vulnerability in admin/plib/api-rpc/Agent.php in Parallels Plesk Panel 7.x and 8.x before 8.6 MU#2, 9.x before 9.5 MU#11, 10.0.x before MU#13, 10.1.x before MU#22, 10.2.x before MU#16, and 10.3.x before MU#5 allows remote attackers to execute arbitrary SQL commands via "unspecified" vectors.

    There is also another vector to be used in Plesk or maybe also other web admin panel like CPanel (noted is not in our case) for this attack. In my case, Plesk webpanel installed in the ISP was targeted since the default configuration permitted the attacker to scan the version of Plesk from the remote by accessing a specific TCP/port. After scan/detecting which Plesk version installed the attacker rapidly atack 250+ servers within range of short time (about a week+), and the infection started in about average time of 10days after attack succeed.

    Moreover in the forensics result of some cases were also detected Wordpress login panel bruted (this is for those sites used WP.. is about 60%+ of the infected of the 250+servers we detected the infection).

    So I presume the hacker has many way to exploit web vulnerability of targeted sites. Meaning, the possibility of the usage of the hack tools/scripts are exist also here. It is my opinion as UNIX engineer: There is no way the human in shell operation can do the infection without the help of automation. really look forwrd to nail the tools used (under investigation).

    The mitigation is (1) To make sure you tune your web admin panel, (2) Make sure you used the latest version of it, (3) If you do not use it just remove the web admin panel service, which I strongly advise, (4) It would be good idea to start to consider applying more secure daemonized services for the vital linux services like with the configuration of SE-Linux or maybe chroot(jailing) the service for the future.

    Regards,

    #MalwareMustDie

    ReplyDelete
  6. After getting reports from users that some weird things were happening on a website that I manage, I started to investigate.
    I used rkhunter, scanned the server locally using clamAV, conducted a remote scan for malware using virustotal, but found nothing.

    New approach ... made a firewall rule that logs all connections made from the server to the internet. After some time (about 15 minutes) an IP-address (217.23.13.65) was logged to which the server made a connection over TCP/80. Okay, was the server checking for updates, or ...? I typed in the ip-address in the google search bar and your article about Darkleech popped up. I Started reading it and investigated some more. There were files in /var/tmp that started with sess_ like mentioned in the article, so I stopped apache, moved all modules to a safe place, reinstalled apache and compared the modules I moved with the new ones. That produced a small list of modules. Looked for them using rpmfind.net and reinstalled the packages of the missing modules that I could find. That narrowed the list down to 5 modules that I still had to investigate.

    Started looking in the apache conf.d directory for clues and could easily strike 4 modules from the list (turned out that the were not used anyway). Only one module was left. The name of the module: mod_string_charset.so. I found that this module was loaded by perl.conf.

    [...]
    # Uncomment this line to globally enable warnings, which will be
    # written to the server's error log. Warnings should be enabled
    # during the development process, but should be disabled on a
    # production server as they affect performance.
    #
    LoadModule string_charset_module modules/mod_string_charset.so
    [...]

    Because I had no clue what it did, I tried to find some information on the internet, but no hits for this module name, but the strings command showed me what I already knew ... this was the module that caused all the problems that the users were experiencing! Removed it, restarted the apache server and everything seems to be OK now. No more sess_-files in /var/tmp, no more connections to that specific IP-address.

    Now that the server is cleaned, updated and extra secured, I hope it stays clean!

    Thank you very much for providing me with enough information so I could solve my problem!

    Best regards.

    Qbeez

    ReplyDelete
    Replies
    1. You narrowed down the possibilities as professional. rkhunter and antivirus is not reliable for the threat like this for lacking of samples shared, believe me I tried them too, but yet in cleaning some servers I must do it mostly manually.

      Let's take this a sign that current *NIX system needs to be monitored, to get more hardening, and don't ever run the default config.

      Stay safe.

      Delete
  7. Thank you so much for fully detailed information about this malware. My website suffered badly because of this malware and the hosting service couldn't do anything about it.

    So I showed them this article and they found the rouge apache module mod_chart_proxy. And fixed this issue. Thank you so much.

    ReplyDelete
  8. Thank you for providing with such detailed analysis and a sample of the malware. I intend to use the malware sample for educational purpose, so can you please provide me with the password of the compressed file.

    ReplyDelete