[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20111109151729.GA18427@openwall.com>
Date: Wed, 9 Nov 2011 19:17:29 +0400
From: Solar Designer <solar@...nwall.com>
To: announce@...ts.openwall.com, john-users@...ts.openwall.com
Subject: John the Ripper 1.7.8-jumbo-8
Hi,
[ Offtopic: anyone with a "permanently failed" MacBook battery (doesn't
charge, shows as "Poor" or "Check battery" in the GUI)? E-mail me
off-list if you don't mind trying a software repair on it (so far tested
on my battery only, with success). No promises, not a free support
offer, ABSOLUTELY NO WARRANTY. And yes, this builds upon Charlie Miller's
research and code. ]
I've just released John the Ripper 1.7.8-jumbo-8:
http://www.openwall.com/john/
Most of the changes since -jumbo-7 are by magnum and JimF - thanks!
My role was mostly limited to preparing the release, although I've also
added a couple of Perl scripts. Also, there's a new Perl script by
Jean-Michel Picod. Here's more detail on the changes:
* Optional OpenMP parallelization has been added for MD5-based crypt(3)
and Apache $apr1$ hashes when building with SSE2 intrinsics, as well as
for SAP CODVN B (BCODE) and SAP CODVN G (PASSCODE). (magnum)
* Raw MD4 has been enhanced with optional SSE2 intrinsics. (magnum)
* The SSE2 intrinsics code for MD4, MD5, and SHA-1 has been pre-built
with Intel's compiler into an assembly file, used with the new
linux-x86-64i, linux-x86-sse2i, and win32-mingw-x86-sse2i make targets
(these i-suffixed targets use gcc and do not require icc to be installed,
yet benefit from the pre-compiled code). (magnum)
* A CRC-32 "format" has been added (see comments and samples in
crc32_fmt_plug.c for proper usage). (JimF)
* Support for occasional false positives or multiple correct guesses has
been added and made use of for WinZip/AES and CRC-32. (JimF)
* "md5_gen", which was not limited to MD5 anymore (also supporting SHA-1
based hashes now), has been renamed to "dynamic". (JimF)
* Two Perl scripts have been added to dump password hashes from Mac OS X
10.7 (Lion) binary plist files - lion2john.pl, lion2john-alt.pl (normally
only one of these is required). (Solar, JimF, Jean-Michel Picod)
* A Perl script has been added to compare two sets of benchmarks (saved
"john --test" output) - relbench.pl. (Solar)
* Numerous other fixes and enhancements have been made, including to
character encodings support and to status reporting (such as to emit a
status line at every cracked password and to show number of candidates
tried in the status line). (magnum, JimF, Solar)
Speaking of the OpenMP parallelization and intrinsics, here are some
benchmarks on 2xE5420 2.5 GHz, gcc 4.5.0, "make linux-x86-64i":
Benchmarking: FreeBSD MD5 [12x]... (8xOMP) DONE
Raw: 215040 c/s real, 26812 c/s virtual
Benchmarking: SAP BCODE [sapb]... (8xOMP) DONE
Many salts: 10166K c/s real, 1267K c/s virtual
Only one salt: 7737K c/s real, 964788 c/s virtual
Benchmarking: SAP CODVN G (PASSCODE) [sapg]... (8xOMP) DONE
Many salts: 6948K c/s real, 867475 c/s virtual
Only one salt: 5061K c/s real, 631062 c/s virtual
Also curious (at least to me, although I am biased) is the benchmark
comparison script, relbench.pl. With 1.7.8-jumbo-8, "john --test"
reports as many as 160 individual benchmark results. Often it is not
clear whether a certain change - different JtR version, compiler,
compiler options, computer - makes things faster or slower overall, by
how much, and how much this varies by hash/cipher type. Although for
your own immediate use you probably care about just one or a handful of
benchmarks, when some of us build John the Ripper packages for others to
use we care about overall performance. Also, being able to compare
different JtR builds like this helps a certain secondary use of JtR: as
a general-purpose benchmark (for C compilers, their optimization vs.
security hardening options, OpenMP implementations, CPUs, computers).
For example, here's how Pentium 3 at 1.0 GHz (gcc 3.4.5, linux-x86-mmx)
compares to one core in E5420 2.5 GHz (gcc 4.5.0, linux-x86-64i, no
OpenMP), both running 1.7.8-jumbo-8:
$ ./relbench.pl 1.7.8-j8-mmx 1.7.8-j8-64i
Number of benchmarks: 160
Minimum: 1.90200 real, 1.90200 virtual
Maximum: 19.68295 real, 19.68295 virtual
Median: 4.08437 real, 4.08439 virtual
Median absolute deviation: 1.25943 real, 1.25940 virtual
Geometric mean: 4.85750 real, 4.85302 virtual
Geometric standard deviation: 1.57265 real, 1.57344 virtual
This shows that speedup varies from 1.9x to 19.7x, with median and
geometric mean at around 4.5x.
Another set of tests shows that gcc 4.6.2's -fstack-protector does not
result in a slowdown, whereas -fstack-protector-all has a 1.2% slowdown:
$ ./relbench.pl asis ssp
Number of benchmarks: 158
Minimum: 0.97152 real, 0.97152 virtual
Maximum: 1.05857 real, 1.06020 virtual
Median: 1.00181 real, 1.00171 virtual
Median absolute deviation: 0.00516 real, 0.00593 virtual
Geometric mean: 1.00311 real, 1.00367 virtual
Geometric standard deviation: 1.01337 real, 1.01360 virtual
$ ./relbench.pl asis sspa
Number of benchmarks: 158
Minimum: 0.91182 real, 0.92094 virtual
Maximum: 1.03674 real, 1.04156 virtual
Median: 0.98883 real, 0.98883 virtual
Median absolute deviation: 0.01139 real, 0.01172 virtual
Geometric mean: 0.98710 real, 0.98765 virtual
Geometric standard deviation: 1.01801 real, 1.01799 virtual
(These are for -jumbo-7 on a Core 2 Duo, no OpenMP.) Of course, with
performance ratios this close to 1.0, you need to run the benchmarks
more than once to make sure the results are reliable.
The relevant concepts are explained here:
http://en.wikipedia.org/wiki/Median
http://en.wikipedia.org/wiki/Median_absolute_deviation
http://en.wikipedia.org/wiki/Geometric_mean
http://en.wikipedia.org/wiki/Geometric_standard_deviation
When interpreting the results, I recommend that we primarily use the
geometric mean, but also look at the median as a sanity check. The
median and the median absolute deviation are more robust when there are
outliers (which the median ignores):
http://en.wikipedia.org/wiki/Robust_statistics#Examples_of_robust_and_non-robust_statistics
http://en.wikipedia.org/wiki/Outlier
On a related note, prompted by a question on john-users, I did some
research on effect of password policies on keyspace reduction:
http://openwall.info/wiki/john/policy
Once in a while, someone posts to john-users asking how to restrict John
the Ripper to match a policy, hoping that this would substantially
reduce the number of candidate passwords to try. Most of the time, such
assumptions are incorrect, although in some extreme cases they are valid.
For example, for printable US-ASCII (95 different characters) and length
8 requiring at least 3 character classes (out of four: digits, lowercase
letters, uppercase letters, and other characters) reduces the keyspace
by only 5.5% (so it is a reasonable thing to do). However, requiring at
least 2 characters of each class (which for length 8 implies exactly 2
characters of each class) reduces the keyspace for length 8 by a factor
of 52.9 (which in terms of keyspace reduction is almost as bad as making
passwords one character shorter) and for length 9 by a factor of 17.6.
Discussion on reddit /r/netsec/:
http://www.reddit.com/r/netsec/comments/lfgbz/effect_of_password_policies_on_keyspace_reduction/
Finally, John the Ripper is still 10th most popular security tool out
there out of 125 top tools per SecTools' latest survey, just like it was
per 2006's survey:
http://sectools.org
http://sectools.org/tool/john/
Along with the update based on the latest survey's results, the
SecTools website has been redesigned to make it more dynamic, so you may
comment on and rate the tools now. (Openwall is not affiliated with
SecTools, but I felt that this SecTools update was worth mentioning.)
Enjoy, and please be sure to provide your feedback on john-users.
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
