[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CADvccEW+WLzPSVPvmUVbgWKNaXcMketcLOiT32=kKmp2AmOL=w@mail.gmail.com>
Date: Thu, 1 Jun 2017 09:03:33 -0500
From: Alex Liu <alexliu0809@...icago.edu>
To: john-users@...ts.openwall.com
Subject: Re: Need help for understanding rule preprocessor
Hi Alexander,
Thank you again for everything! You explanation perfectly resolves my
question. Wish you all the best.
Regards,
*Alex Liu*
alexliu0809@...icago.edu
Computer Science Department
The University of Chicago
On Thu, Jun 1, 2017 at 8:54 AM, Solar Designer <solar@...nwall.com> wrote:
> Hi Alex,
>
> On Wed, May 31, 2017 at 11:40:11PM -0500, Alex Liu wrote:
> > I am a computer science student at The University of Chicago. I am using
> > john (version: 1.7.9-jumbo-7_omp [macosx-x86-64]) and got confused by
> the
> > preprocessor command. I wonder what the following two commands mean:
> >
> > -[:c] <* !?A \p1[lc] p
> > -[:c] l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?\p1[za] \p1[:c]
> >
> > Specifically, what does [:c] and -[:c] mean? What does \p1[za] and
> \p1[lc]
> > mean here?
>
> Since you correctly refer to these pieces as preprocessor commands, it
> appears that you already found they're documented in doc/RULES - but I
> guess the documentation is hard to understand.
>
> What the preprocessor does is expand individual original ruleset lines
> into a possibly larger number of rules. In this case, 2 lines are
> expanded into 4, as you can test and find out in this way:
>
> [user@...t run]$ cat john-local.conf
> [List.Rules:test]
> -[:c] <* !?A \p1[lc] p
> -[:c] l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?\p1[za] \p1[:c]
> [user@...t run]$ echo -n test | md5sum | cut -d' ' -f1 > pw
> [user@...t run]$ echo test | ./john --pipe --rules=test --session=test
> --verbosity=4 --format=raw-md5 pw
> Using default input encoding: UTF-8
> Loaded 1 password hash (Raw-MD5 [MD5 128/128 AVX 4x3])
> Warning: no OpenMP support for this hash type, consider --fork=32
> Press Ctrl-C to abort, or send SIGUSR1 to john process for status
> 0g 0:00:00:00 0g/s 13.33p/s 13.33c/s 13.33C/s tests..Tests
> Session completed
> [user@...t run]$ fgrep -w Rule test.log
> 0:00:00:00 - Rule #1: '-: <* !?A l p' accepted as '<*!?Alp'
> 0:00:00:00 - Rule #2: '-c <* !?A c p' accepted as '<*!?Acp'
> 0:00:00:00 - Rule #3: '-: l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?z :'
> accepted as 'l/a/e/l/o/ssa4se3sl1so0ss$(?z'
> 0:00:00:00 - Rule #4: '-c l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?a c'
> accepted as 'l/a/e/l/o/ssa4se3sl1so0ss$(?ac'
>
> This is with recent bleeding-jumbo. The much older 1.7.9* might not
> have needed the "--verbosity=4" option for this, but otherwise contains
> this logging functionality too.
>
> Here you can see the resulting 4 rules after preprocessor expansion
> (left) and also after processing of the rule reject flags and nop
> squeezing (right).
>
> Now I'll explain what these specific pieces do:
>
> The dashes at the start of lines indicate rule reject flags, quoting
> doc/RULES:
>
> Rule reject flags.
>
> -: no-op: don't reject
> -c reject this rule unless current hash type is case-sensitive
>
> We want the "<* !?A l p" rule to be used unconditionally, but the
> "<* !?A c p" rule used on case-sensitive hash types only. That's
> because using the latter on a case-insensitive hash type would be
> redundant, since the difference between the "l" (lowercase) and "c"
> (capitalize) commands would be lost. For this optimization, we need
> the "-c" rule reject flag on the latter rule.
>
> We also don't want to spend two separate lines on these similar rules,
> so we compact them into one line using the preprocessor. To do this, we
> introduce the "-:" no-op rule reject flag into the first rule and then
> combine the two rule reject flags as "-[:c]". Then we also need to
> combine the "l" and "c" commands, but we want them to be applied along
> with the corresponding rule reject flags. We do this with "\p1[lc]",
> where the "\p1" requests processing "in parallel" with list or range
> "number one", which in this case is our two rule reject flags that are
> the first preprocessor expansion found on this line.
>
> In the second original line, we also use a no-op rule command (on top of
> a no-op rule reject flag), for a similar reason. As you can see, this
> no-op command gets squeezed out and does not affect runtime performance.
>
> Yes, this is way too much complexity for saving two lines, but when you
> get used to these concepts and use them a lot they end up making many
> rulesets many times shorter (and easier to maintain too, as there's less
> duplication so changing something common to multiple ultimate rules
> requires fewer edits).
>
> The below isn't about the preprocessor anymore, but now let's see how
> the rule reject flags actually reject rules:
>
> [user@...t run]$ rm test.log
> [user@...t run]$ echo test | ./john --pipe --rules=test --session=test
> --verbosity=4 --format=LM pw
> Using default input encoding: UTF-8
> Using default target encoding: CP850
> Loaded 2 password hashes with no different salts (LM [DES 128/128 AVX-16])
> Warning: poor OpenMP scalability for this hash type, consider --fork=32
> Will run 32 OpenMP threads
> Press Ctrl-C to abort, or send SIGUSR1 to john process for status
> 0g 0:00:00:00 0g/s 6.666p/s 6.666c/s 13.33C/s TESTS
> Session completed
> [user@...t run]$ fgrep -w Rule test.log
> 0:00:00:00 - Rule #1: '-: <* !?A l p' accepted as '<*!?Alp'
> 0:00:00:00 - Rule #2: '-c <* !?A c p' rejected
> 0:00:00:00 - Rule #3: '-: l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?z :'
> accepted as 'l/a/e/l/o/ssa4se3sl1so0ss$(?z'
> 0:00:00:00 - Rule #4: '-c l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?a c'
> rejected
>
> We requested treating our hash(es) as LM, a case-insensitive hash type.
> As a result, the "-c" rule reject flags actually rejected the two rules
> (out of four).
>
> Alexander
>
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
