The Mossad 2019 Challenge - Part 3

The third challenge begins.

We are given two files: an encrypted file of some sort, and the Windows executable used to encrypt it. The task is simple, reverse engineer the executable and figure out how to decrypt the file.

First off, I’m going to try to encrypt my own file that simply contains “hey” in it, and see what the output would be, to see how the output looks like, compared to the input.

USAGE: Encrypt <input file name> <output file name>
C:\Users\User\Documents\mos>EncryptSoftware.exe secret.txt secret.enc

The program output a file with 3004 bytes. The format doesn’t seem recognizable, let’s go ahead and start reversing.

The main function is easily identifiable due to the usage text we saw earlier. Here’s the pseudo code from Ghidra decompiler, with some variable names I personally assigned:

undefined4 __cdecl main(DWORD argc,int argv)
  LPCWSTR lpFileName;
  DWORD DVar1;
  LPCVOID encrypted_buffer;
  HANDLE *ppvVar2;
  HANDLE createfilehandle;
  DWORD numOfBytesWritten;
  if ((int)argc < 3) {
    printf((int)"USAGE: Encrypt <input file name> <output file name>");
    return 0xffffffff;
  argc = 0;
  encrypted_buffer = (LPCVOID)encrypt_file(*(ushort **)(argv + 4),(int *)&argc);
  lpFileName = *(LPCWSTR *)(argv + 8);
  ppvVar2 = (HANDLE *)HeapAlloc(4);
  if (ppvVar2 != (HANDLE *)0x0) {
    createfilehandle =
    DVar1 = argc;
    *ppvVar2 = createfilehandle;
    if (createfilehandle == (HANDLE)0xffffffff) {
      return 0;
    numOfBytesWritten = 0;
    if (numOfBytesWritten != DVar1) {
  return 0;

The main function doesn’t do too much. Most of the interesting code happens in the encrypt_file function which we’ll go through.

Analysis of the encryption function

The function receives two parameters: one is the path to the input filename, the second is a reference to a variable that stores the buffer size.

The function begins with calling another function, which we’ll call createMD5Hash:


This function gets the length of the input file path, and allocates a buffer of the length plus 6 bytes. So if our file path was aaa.txt (7 bytes), the allocated buffer would be 13 bytes. Then, there’s a call to another function, which calls the GetAdaptersInfo, it returns the MAC address of the first adapter. Finally, you have a file path length + 6 bytes long buffer with the input path, and the additional 6 bytes are the MAC address.

Let’s look at the pseudo code after this:

BVar5 = CryptAcquireContextW(&local_24,(LPCWSTR)0x0,(LPCWSTR)0x0,1,0xf0000000);
if (BVar5 != 0) {
  BVar5 = CryptCreateHash(local_24,0x8003,0,0,&local_1c); // MD5 Hash
  if (BVar5 != 0) {
    BVar5 = CryptHashData(local_1c,(BYTE *)pbData,fileNameLen + 6,0); // Hash the file path + MAC address
    if (BVar5 != 0) {
      BVar5 = CryptGetHashParam(local_1c,2,md5_hash_result,&buffer_size,0); // Output hash into md5_hash_result
      if (BVar5 != 0) {
        output_buffer = (undefined4 *)HeapAlloc(0x20); // Allocate 32byte buffer
        if (output_buffer != (undefined4 *)0x0) {
          offset = 0;
          *output_buffer = 0; // Zero-out the buffer
          output_buffer[1] = 0;
          output_buffer[2] = 0;
          output_buffer[3] = 0;
          output_buffer[4] = 0;
          output_buffer[5] = 0;
          output_buffer[6] = 0;
          output_buffer[7] = 0;
          if (0 < (int)buffer_size) {
            do {
              if (0xf < offset) break;
              // Split each byte to 2 nibbles
              *(byte *)((int)output_buffer + offset * 2) = md5_hash_result[offset] >> 4;
              *(byte *)((int)output_buffer + offset * 2 + 1) = md5_hash_result[offset] & 0xf;
              offset = offset + 1;
            } while (offset < (int)buffer_size);

As seen in the code, the program creates a MD5 hash of the previously created buffer of filename + MAC address. MD5 hashes are 16-byte long, so why does the program allocate a 32 byte buffer? The program actually splits each byte in the MD5 hash into two nibbles (4 bits), and stores each nibble in a byte of the buffer. This can be seen where the program does a shift-right 4 times, and a bitwise AND with 0xf.

This is everything for the MD5 hash creation. Let’s move forward.

After the MD5 calculation, there’s a call to a function that does some sort of AES encryption.

AES Encryption

The function starts by creating a crypto handle using CryptAcquireContextA, the provider type used is PROV_RSA_AES which hints at the program using AES encryption. It then creates a MD5 hash handle, which will be fed data and then passed to CryptDeriveKey. CryptDeriveKey is used to derive and generate a new encryption key from a hash object.

So which data is being given to the MD5 hash? This is important, since if we’re able to derive from the same data, we’ll have the same AES key.

The function allocates a 14-byte buffer, this buffer comprises of 3 things:

  1. 6 bytes MAC address of first adapter (similarly to createMD5Hash)
  2. 4 bytes from the BIOS serial number.
  3. 4 bytes from the disk serial number.

The function gets the BIOS and disk serial number by running wmic bios get serialnumber and wmic diskdrive get serialnumber in cmd.exe, piping the output to a file (command_result.txt) and reading 4 bytes from the second line of the file.

So, the 14 byte buffer is fed to the hash handle, a key is derived and then it encrypts the input file, by block sizes of 16 bytes. The AES encryption func also receives a 2nd parameter where the total encrypted buffer size is stored, this number will be divisible by 16 as 16 is the AES block size.

The junk array, and file format

After the AES encryption part, the program gets 4 bytes from BIOS serial number similar to before, and fills an array of 2500 bytes. The first 4 bytes in that array are the BIOS S/N, and then each 4 bytes are the previous 4 bytes multiplied by 6069, and the last 4 bytes serve as the counter. This junk array plays a role when writing to the file.

This is my python pseudo-code to reconstruct the same array:

import struct
starting_bios_value = bytearray("VMwa") # My VMware BIOS S/N string

def build(reps=623):
  buf = bytearray()
  current_value = struct.unpack("I", starting_bios_value)[0]
  buf += starting_bios_value
  for i in range(reps):
    current_value = (6069 * current_value) & 0xffffffff
    buf += struct.pack("I", current_value)
  # The counter
  buf += struct.pack("I", 624)
  return buf

So the beginning of the file is being built this way:

  1. A 4 byte magic is added (0x531B008A) in the beginning.
  2. The 32 byte MD5 result from createMD5Hash is added.

Then, the AES encrypted data is being written in a very specific way. First the program calculates how much bytes to write out of the AES encrypted buffer each time, and after how many iterations should it drop that number by 1.

bytes_to_write = encrypted_buffer_size / 739 + 1 iterations_until_dropping_write_size = encrypted_buffer_size - (739 * (encrypted_buffer_size / 739))

The loop operates this way:

  1. If the current number of iterations is equal to iterations_until_dropping_write_size, subtract 1 from bytes_to_write.
  2. Write 4 bytes from the junk array.
  3. Write bytes_to_write bytes from the AES encrypted buffer.

After this, the whole buffer goes through a XOR operation before being written to the file. The buffer is XORed against 4 bytes from disk serial number, retrieved in the same method mentioned previously.


To decrypt the file, we will complete the following steps with the encrypted file:

  1. XOR the first 4 bytes with the file magic (0x531B008A), the result is disk serial number.
  2. XOR the rest of the file with the disk serial number.
  3. Rebuild the MD5 and try to find the MAC address. (The filename part is known: intel.txt)
  4. Having the disk serial number, MAC address, and BIOS S/N, we can derive the key, reconstruct the encrypted data and decrypt it.

Solving the XOR

In this following Python script, we XOR the first 4 bytes with the magic to get the disk serial number back.

import struct

FILE_MAGIC = 0x531B008A

def get_disk_sn(first_four_bytes):
  xored_magic = struct.unpack("I", first_four_bytes)[0]
  return xored_magic ^ FILE_MAGIC

input = open("intel.txt.enc", "rb")
output = open("intel.txt", "wb")
disk_sn = get_disk_sn(
print("Disk S/N: " + struct.pack("I", disk_sn))

while True:
    buf =
    if not buf:
    input_byte = struct.unpack("I", buf)[0]
    res = (input_byte ^ disk_sn) & 0xffffffff
    out_byte = struct.pack("I", res)



The output:

Disk S/N: 0000

So now we know the disk serial number, and we have the original file before XOR at intel.txt.

Solving the MAC address

Now it’s important to use a hint that we’ve been given at the beginning of the challenge. The challenge text mentioned the manufacturer was “Or… Po… Ltd.”, so we’ll look for this specific MAC vendor and it’s MAC address range.

I personally downloaded the MAC vendor list from GitHub, and ran a regex with Sublime Text: Or.+.+.+ P. I found this MAC vendor with the 00:13:37 address prefix: 001337 Orient Power Home Network Ltd.

So now we know the first 3 bytes of our MAC, it’s a matter of a few seconds to brute-force through 0xffffff MD5 combinations. I wrote a Python script to reconstruct the MD5 from the nibbles and brute-force the MD5 hash to find the rest of the MAC:

import itertools
import hashlib
import struct
import binascii

known_filename = "intel.txt"
known_mac = "\x00\x13\x37"

input = open("intel.txt", "rb")
packed_md5 = bytearray(
real_md5 = bytearray()

# Reconstruct MD5 from nibbles
for i in range(16):
  real_md5.append((packed_md5[i * 2] << 4) | packed_md5[i * 2 + 1])

print("found md5")

# bruteforce
print("bruteforcing 3 bytes")
allbytes = [chr(c) for c in range(256)]
for byte_sequence in itertools.permutations(allbytes, 3):
  if hashlib.md5(known_filename + known_mac + "".join(byte_sequence)).digest() == real_md5:

And we got our 3 bytes after a few seconds, the MAC is 00:13:37::8e:ab:66:

found md5
bruteforcing 3 bytes
('\x8e', '\xab', 'f')

As for the BIOS serial number, I was working in a VMware Windows machine and my encrypted files matched the bytes where I guessed the BIOS serial number was supposed to be retrieved from. So personally, I didn’t need to write any scripts, I just knew the 4 bytes were “VMwa”, Lucky me.

To solve this, I wrote a Python script that utilizes wincrypto, which provides WinCrypto bindings for Python. I have followed the same calculation as they are in the binary, the script can be executed against the file after reversing the XOR operation with the previous script.

import struct
from wincrypto import CryptCreateHash, CryptHashData, CryptDeriveKey, CryptDecrypt
from wincrypto.constants import CALG_MD5, CALG_AES_256
import os.path

known_mac = "\x00\x13\x37\x8e\xab\x66"
known_serial = "0000"
target_file = "intel.txt"

assumed_filesize = (os.path.getsize(target_file) - 2992)
encrypted_size = assumed_filesize

if encrypted_size % 16 != 0:
  encrypted_size += (16 - (encrypted_size % 16))

print("encrypted_size: %d" % encrypted_size)

bytes_written_each_time = encrypted_size / 739 + 1
repetitions_until_dropping_write_size = encrypted_size - (739 * (encrypted_size / 739))

infile = open(target_file, "rb")

magic =
if struct.unpack("I", magic)[0] != 0x531B008A:
  print("magic mismatch")
  raise IOError

print("magic match! :)"), 1) # Offset 32 bytes ahead to skip MD5

encrypted_buf = bytearray()
counter = 0
while len(encrypted_buf) != (encrypted_size - 4):
  if counter == repetitions_until_dropping_write_size:
    bytes_written_each_time -= 1, 1) # Ignore 4 bytes from the 'junk array'
  encrypted_buf +=
  counter += 1

encrypted_buf = str(encrypted_buf)
encrypted_buf += "\x00"*4 # Pad to match blocksize

hasher = CryptCreateHash(CALG_MD5)
CryptHashData(hasher, known_mac + known_bios_sn + known_serial)
aes_key = CryptDeriveKey(hasher, CALG_AES_256)
decrypted_data = CryptDecrypt(aes_key, encrypted_buf)
print("len of decrypted data: %d" % len(decrypted_data))

output = open('solution.txt', 'wb')

Let’s execute the script and look at the contents..

encrypted_size: 35936
magic match! :)
len of decrypted data: 35872
C:\Users\User\Documents\mos>type solution.txt

And voilà:

Yanir Tsarimi

Yanir Tsarimi

Security enthusiast, developer, and blogger (sometimes)

comments powered by Disqus
rss facebook twitter github gitlab youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora