How to Manage Big Passwordlists

In the penetration testing journey we must learn how to brute force attack. But the success of brute force attack depends on our password list. We need a bigger and specialized password list to be successful in dictionary attack or brute force attack. We can find over 20GB password lists on the internet easily.

Managing a bigger sized password list is always a headache for everyone. Ever tried to edit an <10GB wordlist text file on a mid-range laptop? After trying this anyone will know what is real pain!

But sometimes we need to create a custom wordlist and we also want to use the big sized common password list. Sometimes we also need to delete some passwords from the list or add some passwords to the list. In this situation we have two ways to do it.

First is we may use a famous tool called Dymerge. This tool might be famous but we think it sucks. Then we should follow the second one. Wait a bit first we are going to learn about Dymerge tool then we learn about the second method.

Dymerge in Kali Linux

Dymerge is a python tool that we can use our password lists. It is a simple command line tool that help us to marge wordlists or add/remove specific keywords from a wordlist.

Let's use Dymerge in our Kali Linux system. First we need to copy it from it's GitHub repository by using following git clone command:

git clone https://github.com/k4m4/dymerge

The screenshot of the command is following:

Then we need to navigate inside the dymerge directory by using following command:

cd dymerge

We can see following files in dymerge directory by using ls command:

ls

In the following screenshot we can see the dymerge.py the python script we need.

To see Dymerge's help option we can use following command:

python dymerge.py --help

Dymerge will open the help menu after the above command as the following screenshot:

Here we can see that we can marge two wordlists using this tool. For example we gathered some wordlist files on our Desktop.

Now we will marge all these wordlists into a giant wordlist and we want the output in compressed format. Here we compress all the wordlists on the directory, for that we just need to specify the directory only.

If we want to merge individual wordlists then we have to specify the file names also. -o flag is used for the output location & format. We also want to insert one more password "KaliLinuxIn" into our giant wordlist, so use need to use -i flag.

We can remove the duplicate passwords and keep the unique passwords only by just using the -u flag. We also can reverse the passwords by using -r flag but we are not using that on our this example.

So we apply the following command in our terminal.

python dymerge.py /home/kali/Desktop/my-wordlists -u -f -o /home/kali/Desktop/my-wordlistsgiantwordlist.txt -i KaliLinuxIn -z bz2

Here we have run the command and the output is following:

In the above screenshot we can see Dymerge is started its job. We just need wait some seconds (It's depends on our wordlist's size and our machine's processing power). After finish the work it looks like the following screenshot:

We had choose the same directory for the giant wordlist's output as zip. After complete the process we got the giant wordlist on our required directory or folder, as we can see in the following screenshot.

We got the giant zipped wordlist. Don't follow our wallpaper it changes automatically in minutes

That is how Dymerge works. But this can be replaced by some simple bash commands that is the second method. Bash can easily replace all it's functions, so why we need this? That why we told that Dymerge sucks.

DyMerge Vs Bash

When Bash commands can replace Dymerge then why we need to carry an extra python script on our machine. We don't need to download and run an extra tool to do it.

Moreover, Dymerge doesn't work well with too large files. This is because it loads every word from each dictionary into an array, causing the computer system's memory to overload and possibly even crash.

Here is some Dymerge uses and Bash commands that can replace those:

Dymerge command:

python dymerge.py /usr/share/wordlists/rockyou.txt /lists/cewl.txt -s -u

Bash command to replace this

sort -u /usr/share/wordlists/rockyou.txt /lists/cewl.txt > output.txt

Dymerge command:

python dymerge.py /lists/cewl.txt /lists/awlg.txt -s -u -i Passw0rd,inpuT

Bash command to replace this

sort -u /lists/cewl.txt /lists/awlg.txt <(echo 'Passw0rd\ninpuT') > output.txt

Dymerge command:

python dymerge.py ~/fsocity.dic -s -u -r -o ~/clean.txt

Bash command to replace this

sort -r <~/fsocity.dic | uniq> clean.txt

Dymerge command:

python dymerge.py /dicts/crunch.txt /dicts/john.txt -s -u -f -z bz2

Bash command to replace this

sort -u /dicts/crunch.txt /dicts/john.txt | bzip2 > output.bz2

Even though all of DyMerge's main features can be fulfilled with classic bash, the tool has actually gained publicity.  -- Nikolaos Kamarinakis (Creator of DyMerge)

So, this all for this tutorial hope everyone liked this. In this tutorial we have learned how to manage bigg wordlists on our system.

Love our articles? Make sure to follow us to
get all our articles directly on inbox. We are also available on Twitter and GitHub, we post article updates there. To join our family, join our Telegram Group. We are trying to build a community for Linux and Cybersecurity. For anything we always happy to help everyone on the comment section. As we know our comment section is always open to everyone. We read each and every comment and we always reply.