This is a part of a series of writeups for a malware challenge I made for DamCTF 2020. Please see here for the overview.

Phase 3

Great work extracting the config from that malware! Based on your analysis, we’ve identified many other samples for another variant that appears to be using a slightly different config encryption function. Can you automate the config extraction for these new samples?

Please connect to our sample server to receive samples for analysis: nc 32153

Now that players knew where the malware config was located and generally how to extract the flag, they would need to automate that extraction process to provide a specified config item from 10 different binaries within 60 seconds. The “sample server” had 3000 different binaries that had randomly generated configs. However, we threw a curveball at players by modifying the config encryption code for the phase 3 binaries, so players would have to save the first binary for manual analysis to build their solving script.

When players connect to the service, they are given a Base64 blob of the ELF to analyze, and then are prompted to provide a config value:

What's the value of the 'xvee' config item?

Assuming they provide the correct config value, they will repeat the process 10 times and will then be given the flag.

Let’s take a look at the new config decryption code from one of the binaries:

void FUN_00101a19(uchar *param_1,long param_2)

  char *__src;
  uchar *outdata;
  long in_FS_OFFSET;
  byte local_455;
  int local_454;
  int local_450;
  RC4_KEY local_438;
  byte local_28 [24];
  long local_10;
  local_10 = *(long *)(in_FS_OFFSET + 0x28);
  __src = (char *)(param_2 + 1);
  outdata = (uchar *)calloc(0x42,1);
  strncpy((char *)(outdata + 0x10),__src,0x31);
  local_454 = 0;
  while (local_454 < 0x31) {
    RC4(&local_438,0x10,outdata + local_454,local_28);
    local_455 = 0;
    local_450 = 0;
    while (local_450 < 0x10) {
      local_455 = local_455 ^ local_28[local_450];
      local_450 = local_450 + 1;
    __src[local_454] = __src[local_454] ^ local_455;
    local_454 = local_454 + 1;
  if (local_10 != *(long *)(in_FS_OFFSET + 0x28)) {
                    /* WARNING: Subroutine does not return */

FUN_00101a19 in Phase 3 uses RC4-CFB8, which for some reason is a supported crypto scheme in OpenSSL but I’ve been told should never be used and is effectively a meme (dear crypto people: I am not a crypto person, and apologize in advance). One of OSUSEC’s resident crypto experts helped with the specific crypto implementations for phases 2 and 3, otherwise it would’ve just been random XOR if I had implemented it (thanks Athos!).

Guide to solving beta version of Phase 2/3

Guide to solving beta version of Phase 2/3

Using the above decompilation, we can write a new function in Python to decrypt a config chunk, along with the other code necessary to interact with the service:

from pwn import *
import sys
from Crypto.Cipher import ARC4
import base64
from typing import Dict

def get_config(elf: bytes) -> Dict[str, str]:
    def decrypt_chunk(key, chunk):
        cipher =
        outbytes = list(chunk[1:])

        ret = cipher.decrypt(key[:16])
        ptr = ret + chunk[1:] 

        for i in range(49):
            ret = cipher.decrypt(ptr[i:i+16])
            local_455 = 0
            for j in range(16):
                local_455 ^= ret[j]
            outbytes[i] ^= local_455

        return bytes(outbytes)

    config_bytes = elf[0x51a0:0x5394]
    config_key = elf[0x4010:0x4010+32]

    config = {}

    for i in range(10):
        plaintext = decrypt_chunk(config_key, config_bytes[i*50:(i+1)*50])
        key = plaintext[:4].decode()
        val = plaintext[4:].decode().split("\x00")[0]
        config[key] = val
    return config

p = remote(sys.argv[1], int(sys.argv[2]))

p.sendlineafter("continue)", "")

for _ in range(10):
    a = p.recvline()
    elf = base64.b64decode(p.recvline())

    config = get_config(elf)
    line = p.recvline()
    key = line.decode().split("'")[2][:4]



Running that script will grant us the Phase 3 flag after ~15 seconds:

$ python3 32153
[+] Opening connection to on port 32153: Done