Custom Encoding

Malware often uses custom encoding schemes to conceal their payloads and avoid detection. These custom schemes can be full custom layers, or they can be variations of known algorithms such as XOR or Base64. Using custom encoding schemes allows malware to encode their payloads in a unique way that can be difficult for security tools and forensic investigators to detect.

In some cases, the custom encoding scheme may be a combination of different algorithms, such as using XOR to encrypt the payload and then using Base64 to encode the resulting ciphertext. This can make the encoded payload even more difficult to detect and analyze.

To safeguard payloads and communication channels, multiple data encoding operations (e.g., XOR, ADD, SUB, ROL, and ROR) can also be chained together with a randomly generated key. These operations are not only simple to execute and reverse but also demand minimal computing resources.

Technique Identifiers

U0702 E1027.m03

Technique Tags

Custom encoding schemes Conceal Payloads Full custom layers Variations Known algorithms XOR Base64

Evasion Categories

Data Obfuscation

Code Snippets

Description

The original message is first encrypted using a custom XOR encryption algorithm. The encrypted message is then encoded using a custom Base64 algorithm. The resulting encoded message is then printed to the screen. This encoded message can then be decrypted and decoded using the same custom algorithms in order to access the original message. This example demonstrates how a malware author could use custom encoding schemes to conceal their payloads.

# Define the original message to be encoded
message = "Hello, world!"

# Encrypt the message using a custom XOR encryption algorithm
encrypted_message = ""
for i in range(len(message)):
  encrypted_message += chr(ord(message[i]) ^ 0x5)

# Encode the encrypted message using a custom Base64 algorithm
encoded_message = ""
base64_chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
for i in range(0, len(encrypted_message), 3):
  b1 = ord(encrypted_message[i])
  b2 = ord(encrypted_message[i + 1]) if i + 1 < len(encrypted_message) else 0
  b3 = ord(encrypted_message[i + 2]) if i + 2 < len(encrypted_message) else 0

  c1 = b1 >> 2
  c2 = ((b1 & 0x3) << 4) | (b2 >> 4)
  c3 = ((b2 & 0xf) << 2) | (b3 >> 6)
  c4 = b3 & 0x3f

  encoded_message += base64_chars[c1] + base64_chars[c2] + base64_chars[c3] + base64_chars[c4]

# Print the resulting encoded message
print(encoded_message)

About Author Open Snippet

Author: Thomas Roccia (fr0gger) / Target Platform: Windows

Description

This script encodes a given plaintext shellcode using a combination of XOR, ADD, SUB, ROL, and ROR operations with a randomly generated key. The purpose of the encoding is to make the shellcode more difficult to detect and analyze.

#!/usr/bin/env python3
# Original Source: https://github.com/wand3rlust/Niernen

import random

#Encode shellcode using XOR, ADD, SUB, ROL, and ROR
def encode_shellcode(shellcode, key):
    encoded_shellcode = bytearray()
    for i, byte in enumerate(shellcode):
        #XOR
        xored_byte = byte ^ key[i % len(key)]
        #ADD
        added_byte = (xored_byte + key[(i + 1) % len(key)]) % 256
        #SUB
        subbed_byte = (added_byte - key[(i + 2) % len(key)]) % 256
        #ROL
        rolled_byte = ((subbed_byte << 1) | (subbed_byte >> 7)) % 256
        #ROR
        ror_byte = ((rolled_byte >> 1) | (rolled_byte << 7)) % 256
        encoded_shellcode.append(ror_byte)
    return bytes(encoded_shellcode)


#Generate a random key of given length and convert it into bytes
def generate_key(length):
    return bytes([random.randint(0, 255) for i in range(length)])


plaintext_shellcode = input("\nEnter plaintext shellcode: ")
#Encode the user unput into UTF-8 and change from string to byte
shellcode = plaintext_shellcode.encode()
#Generate same length key as shellcode hex
key = generate_key(len(shellcode))
#Call encode_shellcode function with 2 arguments i.e, UTF-8 shellcode and key
encoded_shellcode = encode_shellcode(shellcode, key)
print("\nOriginal shellcode (in hex): ", shellcode.hex())
print("\nKey (in hex): ", key.hex())
print("\nEncoded shellcode (in hex): ", encoded_shellcode.hex())
#Convert byte format to string
encoded_shellcode = encoded_shellcode.hex()
#Append \x after every 2nd character
encoded_shellcode = "\\x" + "\\x".join(encoded_shellcode[i:i + 2] for i in range(0, len(encoded_shellcode), 2))
print("\nEncoded shellcode (with \\x): ", encoded_shellcode)
print("\n")

About Author Open Snippet

Author: Abhijeet Kumar / Target Platform: Windows

Additional Resources

External Links

The resources provided below are associated links that will give you even more detailed information and research on current evasion technique. It is important to note that, while these resources may be helpful, it is important to exercise caution when following external links. As always, be careful when clicking on links from unknown sources, as they may lead to malicious content.

An Example of Common String and Payload Obfuscation Techniques in Malware

Created

March 18, 2019

Last Revised

March 24, 2026