Programming stuff

Yesterday I finished reading Scott Rosenberg's book Dreaming in Code - Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software (Official website / Wikipedia). Inspired by a severely problematic content management system for salon.com, Rosenberg started to think about the problems of creating complex software. Why do so many software projects fail to meet deadlines? Why do they spiral off into money black holes? Not being a software developer himself, the journalist Rosenberg started to hang out with the developers of Chandler (Official website / Wikipedia), a "revolutionary" personal information manager. He followed the Chandler team for three years, attended meetings and presentations, and asked the people involved in the project about their thoughts about software development in general and Chandler in particular. The results of this journey are written down in the book.

Continue reading "Book Review - Dreaming in Code"

Yesterday I saw a talk given by Frank Boldewin where he mentioned the FreeIconList trick to fool code emulators. At this point I started to wonder what other Win32 API functions are basically unused. Using Ero Carrera's Python library pefile to parse PE files I wrote a small Python script that tries to find out what Win32 API are basically unused.

The modus operandi was simple. I read the exported functions of all DLL files in WindowsDir and WindowsDir/system32 and compared them to the functions imported by all EXE/DLL files in WindowsDir, WindowsDir/system32 and my entire Program Files directory.

The first result is that most exported functions are apparently basically never used. My script managed to find 127569 exported functions in 1225 DLL files. 104608 of those are never used by the 6615 EXE/DLL files which import functions ("used" is liberally defined as "imported through the import directory" here, of course). That leaves 22961 functions which are actually used.

Here are some output files which show the exported DLL functions sorted by their usage. The numeric column contains the number of PE files which import the function statically. That means that 3475 of the 6615 files I tested import GetLastError for example.

• Click here to see the Top 2000 most used API functions
• Click here to see the usage statistics of all advapi32.dll functions
• Click here to see the usage statistics of all gdi32.dll functions
• Click here to see the usage statistics of all kernel32.dll functions
• Click here to see the usage statistics of all msvcrt.dll functions
• Click here to see the usage statistics of all ole32.dll functions
• Click here to see the usage statistics of all oleaut32.dll functions
• Click here to see the usage statistics of all shell32.dll functions
• Click here to see the usage statistics of all user32.dll functions

Random notes

• kernel32.dll is surprisingly dominant while gdi32.dll is surprisingly "unused"
• pefile is extremely awesome and easy to use
• Don't be confused that API functions like lstrlen are imported 0 times, check lstrlenA and lstrlenW

Click here to download the Python script.

Hotch 1.0.0 - named after everyone's favourite TV profiler - is an IDA plugin that can be used to profile binary files. It sets breakpoints on all basic blocks of a program, records breakpoints hits and tries to figure out statistics from these hits. Click here to seen an example of a simple profiling session (starting Notepad and exiting Notepad again). Click here to see a huge 6.5 MB results file that shows a larger profiling session (loading a file in Notepad and playing around in it).

Random Notes:

• "This is really slow for larger files". Yeah, it is really slow in IDA up to 5.2 but Ilfak fixed some things in IDA 5.3 and it works acceptably fast now. So patience, young padawan.
• "The timing results don't really make sense". Yeah, I know. Since I execute a callback function after each breakpoint hit tight loops take disproportionally much time. For anything but tight loops the timing results should kinda work, at least relative to each other of course.
• Ignore the source file libida.hpp, it's an early version of my experimental-at-best C++ wrapper library for the IDA SDK.
• I take feature requests for Hotch.

Click here to download Hotch 1.0.0 (full source included, of course).

Here's version 1.4.0 of my hex editor Hexer. I fixed all reported bugs and added most of the feature requests I've received so far. The following stuff changed:

New features

• Core: Added the option to reload plugins
• Core: Improved color maps
• Core: Added the option to change the font size in hex windows
• Core: Added the option to move the hex window tabs to the right side
• Core: Added a menu option to insert an array of bytes
• Core: Hex windows now have context menus when the user right-clicks on the window
• Core: Added the export options (like Export as C++ Array) to the hex window context menu
• Core: Added support for file address bookmarks
• Plugins: Added a new JRuby example plugin (BitmapStructure.rb)

Bugfixes

• Core: Fixed a bug that caused nullpointer exceptions when inserting bytes into new/empty files
• Core: Fixed a bug that caused structure highlighting to disappear when the structure was partially scrolled off the screen

Changes to the Plugin API

• Added the option to remove menus from the Tools and Edit menu (IMainWindow)
• All plugins must implement the unload method from now on (IPlugin)
• Added the pluginInitialized method to IPluginListener
• Added the IColormap interface for improved colormap support
• Changed the IColormapPlugin to make color maps easier to use and more powerful
• Added the option to change the position of the tabs in hex windows (IHexWindow)
• Added the setFontSize method to IHexWindow
• Added the IHexPopupPlugin interface which can be used to extend the context menus of hex windows
• Added the option to change the size of a hex window (IHexWindow)
• Added the option to change the relative sizes of the hex view and the tabs in IHexWindow

The coolest new feature is probably the option to reload all loaded plugins. This is very useful when developing scripts or plugins. Just hit ALT-SHIFT-R and all plugins in the plugins and scripts directories are reloaded. That means you do not have to restart Hexer anymore while testing new plugins.

The option to move the tabs in hex windows (like structure viewer, bookmarks, file stats, ...) to the right side of the hex window turned out pretty cool too. Saves lots of space on the screen.

Anyway, you can download Hexer 1.4.0 here. I'm happy to hear of any bug reports or feature suggestions.

In Hexer the Structure Viewer is the part of the hex windows where data from loaded files is read into pre-defined structures and displayed in trees that represent the structure. This is pretty useful because you can use it to quickly navigate through complexely structured binary data.

Like basically all parts of Hexer you can extend the Structure Viewer tree with plugins that can be implemented in Java, Groovy, ECMAScript, Python, and Ruby. I have prepared an example plugin (download here) which demonstrates how to extend the Structure Viewer tree using a plugin written in Ruby. This plugin adds support for Windows Bitmap file (BMP) headers. The result can be seen in the screenshot below.

Note that this plugin only works if you have Ruby scripting support installed in Hexer.

Here's my hex editor Hexer 1.3.0 which brings the following new features:

• Core: Added support for different languages (English and German are available so far)
• Core: Tools and components that are closely tied to individual files (file statistics, structure viewer, ...) are now part of the hex windows of those files.
• Core: A few commandline options were added.
• Core: Console mode was added to the scripting dialog.
• Core: Added scripting support for JRuby

The new console mode for scripting is actually pretty cool. It is very helpful when you're developing scripts or just toying around with the scripting API for a while (enter as many lines in the console as you want and then hit CTRL-ENTER to execute them). Thanks to a recently fixed bug in the JRuby scripting engine Hexer can now finally support Ruby scripting too for those who prefer Ruby over Python.

Some bugs were fixed too:

• Core: Added missing tooltips to several menus.
• Core: The Java-Array exporter now creates a byte[] array instead of a char[] array
• Core: Fixed a bug which broke shell integration when Hexer was placed in a directory that contains whitespaces.

I've also made some changes to the Plugin API

• Added the option to display simple progress dialogs.
• For performance reasons, the method IHexFile::getData does not return a copy of the file data anymore but the actual file data itself.
• Several interfaces that deal with the structure viewer were added or modified.
• The IStatsPlugin was replaced by the IFileSpecificPlugin

You can download Hexer 1.3.0 here. Any kind of feedback is appreciated. Thank you.

Unless something goes spectacularly wrong I'll be in Montreal from next Thursday to Sunday to attend RECON 2008. If you wanna do one or more of the following things, make sure to meet me at some point:

• Talk/Complain about BinNavi
• Talk/Complain about BinDiff
• Talk/Complain about VXClass
• Talk/Complain about Hexer
• Talk/Complain about Zynamics
• Talk/Complain about any other topic
• Get an awesome VXClass t-shirt
• Have some poutine with me

I'll be the guy with the yellow Sabre Security bag. Alternatively you can shoot me an email or ask around.

After my last two book reviews were rather negative, I'm happy to say that this review is going to be positive again. Mario Herwardt's and Daniel Pravat's book Advanced Windows Debugging (Official Website / Amazon) keeps what the title promises. It's a book for people that need to find bugs in Windows programs that are for whatever reason (too) difficult to find with the "normal" developer tools like the integrated Visual C++ debugger. That's not what the authors say in the introduction of course. They say it's a book for everyone that does Windows development. And that's probably true because the book is a real eye-opener for what's possible with the debugging tools provided by Microsoft. Nevertheless I guess that most developers will probably rarely if ever leave the cushy environment of their IDE's debugger. But just in case you do, this book prepares you well. Continue reading "Book Review - Advanced Windows Debugging"

The 1.2.0 release of Hexer improves the usability of Hexer. On Windows, Hexer can now be integrated into the context menu that appears when you right-click on files in the Windows Explorer (and elsewhere). Furthermore the main window of Hexer is now scrollable. This gives the users extra space and allows him to open more windows. Another nifty thing is that the scripting dialog now supports syntax highlighting and other small features that make it easier to write scripts.

You can download Hexer 1.2.0 here.

Complete list of changes:

New Features

• Core: The main window now contains scrollbars. This gives the user more space for open MDI windows.
• Core: A tool bar that shows buttons for all open windows was added to the main window. This makes it easier to quickly switch to another window.
• Core: Progress bars are shown during long operation like plugin loading, file loading, or script execution.
• Core: The Windows menu shows a list of open windows for the user to switch to.
• Core: Added Cascade/Tile Windows feature in the main window
• Core: User is asked whether he wants to make a backup of a file before saving it.
• Core: It is now possible to stop running scripts that are stuck in infinite loops or simply take too long to execute.
• Core: Added optional Windows Explorer integration. Hexer can now be started by right-clicking on files.
• Core: Improved the scripting dialog (added syntax highlighting, tab handling, undo and redo, cut/copy/paste, icons, and other things)

Bugfixes

• Core: Problems with the split windows in the settings and plugin dialog were fixed.
• Core: Improved the layout of the hex window status bar.
• Core: Fixed a few problems during plugin initialization and loading.

Changes to the Plugin API

• All MDI windows added to the MainWindow must now extend the class HexerWindow
• Added the class HexerWindow which is the base class of all MDI windows of the main window
• Added the methods tileWindows and cascadeWindows to the IMainWindow interface
• Added the method removeChildWindow to the IMainWindow interface

Screenshots of the new scripting window and a new screenshot of the main window:

This week I managed to read Reverse Engineering Code with IDA Pro. I was pretty curious about the book because it's the first book specifically about everyone's favourite disassembler IDA Pro and it turned out to be very different from what I expected.

Co-authored by Dan Kaminsky (editor), Justin Ferguson, Jason Larsen, Luis Miras, and Walter Pearce, Reverse Engineering Code with IDA Pro sets out to give an introduction to IDA Pro and how to use it to reverse engineer software. The book is approximately 310 pages long and divided into nine chapters ("Introduction", "Assembly and Reverse Engineering Basics", "Portable Executable and Executable and Linking Formats", "Walkthroughs One and Two", "Debugging", "Anti-Reversing", "Walkthrough Four", "Advanced Walkthrough", "IDA Scripting and Plug-Ins").

Continue reading "Book Review - Reverse Engineering Code with IDA Pro"

Hexer 1.1.0 (click here to download) is primarily a bugfixing release. The following things changed since Hexer 1.0.0:

New Features

• Core: Added a status bar that displays the current offset in hex windows
• Core: Support for the ENTER key to execute a dialog's default action was added to all dialogs
• Core: When selection a result of a file comparison, the corresponding area is highlighted in the hex window
• Core: Added options for configuring the appearance of the hex windows
• Core: Structure Viewer was sped by only calculating the values of visible structure elements
• Plugins: Added hotkey management to avoid duplicate hotkeys
• Plugins: Added a sample plugin that shows how to write plugins in Java
• Plugins: Plugin CLASS and JAR files can now be placed in subdirectories of the plugins directory

Bugfixes

• Core: Improved Linux support
• Core: Fixed a problem with copy/pasting
• Core: Fixed a problem with incorrect highlighting in dialogs like Compare Files
• Core: Structure highlighted is removed when the Structure Viewer dialog is closed
• Core: Fixed some problems when searching beyond the end of files
• Core: Existing dialogs are brought to front when their menu hotkey is pressed again
• Plugins: Fixed a problem that prevented plugin JAR files from being loaded

Changes to the Plugin API

• Added the option to insert bytes into IHexFile objects
• Added the registerAccelerator method to IHexWindow which helps to make menu accelerators unique
• Added finer-grained methods for watching hex windows to IMainWindowListener
• Added several methods for configuring the appearance of hex windows to IHexWindow
• Added a clear method to IStructureHighlighter which clears previous data highlighting
• Added the IStructureViewerHelper which can be used by data viewer plugins to figure out whether they are visible in the Data Viewer dialog
• Moved the method addChildWindow from IPluginInterface to IMainWindow

I hope Hexer works on Linux now. At least it does work on my Ubuntu.

Please report bugs and request features by replying to this blog entry or by sending me an email (see the Contact information in the docs directory of the RAR file).

I finally got around to write an example plugin for my hex editor Hexer to show how simple it is to extend Hexer according to your own needs. The Java plugin I am going to present calculates the entropy of files according to the method presented on Ero Carrera's blog. The plugin adds a new tab containing a line chart and a button to the File Statistics dialog. When the user clicks the button, the entropy of the active file (that is the file in the last active hex window) is calculated and shown in the line chart. The screenshot below shows the entropy distribution of Notepad.exe.

You can download the source file of the plugin here. The archive contains the source file EntropyCalculator.java as well as two class files which were created by compiling the source file using Java 1.6. To install the plugin, simply copy the two class files to the plugins directory of your Hexer installation. Since the plugin uses the JFreeChart library to display the graph it is also necessary to get the files jcommon-1.0.12.jar and jfreechart-1.0.9.jar from the JFreeChart package. Copy those files into the jars directory of your Hexer installation.

Continue reading "Sample Hexer Plugin: Calculating the entropy of a file"

Hi everyone and welcome to another post in my favourite blog entry category: Book Reviews. I'm happy to announce that for the first time ever I have actually managed to read a book and write a review of it before its official release (unlike my other reviews where I often review three year old books). I'm talking about Adam Shostack's and Andrew Stewart's new book The New School of Information Security here which will be released tomorrow.

The New School of Information Security is a weird book. From the title of the book you'd think that this is a book about information security for people who have at least some kind of clue about information security. I mean why would people that do not have a clue about information security read a book about reforming and improving the field of information security? Unfortunately this assumption is wrong.

Continue reading "Book Review - The New School of Information Security"

Just a few days later than originally announced, I managed to finish the first release of my hexeditor Hexer. It's available for download here. Please report any problems or suggestions by replying to this blog entry.

The coolest feature of Hexer is the scripting and plugin support. Note that Hexer only supports ECMAScript out of the box. If you want Python or Groovy scripting support, you need to install and download the necessary Jython/Groovy packages. Please read the manual in the docs folder to find out exactly what to download.

There are sections in the manual that describe how to write plugins and scripts. Example scripts are included in the RAR file. There are no example plugins yet but I'll write something about them on my website before next weekend.

Update:

Known issues:

• Doesn't work on Linux

Remember Hexer? It's back. In Java form. For those who don't remember, back in 2005 I posted the alpha version of Hexer, a hex editor written in C#. Like most of my projects, Hexer was abandoned for lack of time. However, the basic idea of an easily extensible hex editor still appeals to me and so I decided to bring it back.

The new version of Hexer is not written in C# anymore. I ditched C# for Java for practical reasons. The primary reason is that I'm using Java at work which means I'm doing nearly all of my development these days in Java. The second reason is that we've had lots of code that's necessary for Hexer in our company's internal Java library already and I only had to combine our Java library with some new code. That way I managed to implement the first Java version of Hexer in a very short amount of time.

Alright, if everything works out smoothly the first version of Hexer 1.0.0 will be released next week (depending on how fast I can write the remaining unit tests, the documentation, and how quickly I can get someone to update the company website where Hexer will be available as a free download). So let's have a sneak preview of Hexer 1.0.0.

Continue reading "Hexer 1.0.0"