The Code Cave

AKA: Alternate Data Streams – An introduction…

One of the members of a technical forum to which I belong had a virus detected in a dll that he could not find anywhere on his computer.  In the end, it was discovered that the file was embedded as Alternate Data Stream (ADS) attached to his System32 directory.  The full path to the file was something simlar to c:\windows\system32:dhht.dll.

I was not familiar with using streams in this manner; so I looked it up.  In short, any file or directory on an NTFS drive can any number of files piggy-backed onto it without affecting its reported file size.  These files remain intact through normal file copying and renaming actions as long as the file remains on an NTFS drive (or compatible archiver).

I’d never come across this before and thought I would share.
Summary/Demo….    (adjust paths as necessary)

1. Create a directory off the root called test
   From the command prompt in your test directory type:
      [DOS]Echo a>test.txt[/DOS]
      That will create a nice 3 byte text file.
2. Now type:
      [DOS]type c:\Windows\system32\notepad.exe>test.txt:notepad.exe[/DOS]
3. Get another directory listing and notice that the size of test.txt remains 3 bytes though an additional ~70kb has been added to it.  The time has changed but that could be set back.
4. Now, from the command prompt in your test directory, execute the hidden stream:
      [DOS]start \test\test.txt:Notepad.exe[/DOS]
      (You must use include the path to test.txt)

You are now running an exe that was hidden in 3 byte text file!
As far as Windows is concerned, that file takes up only enough space for three characters.  Yet it’s got this hidden back end on it.  That txt file could be taking up any amount of space.  Perhaps it is storing 5 gigs of pictures in it.  Who knows?
Windows supplies no tools to display this.  The zip file referenced in this article, includes an exe that does. 
The Task Manager shows the running proces in different ways depending upon your Windows version.  In SP2, the executable is listed as test.txt:Notepad.exe. However, in earlier versions of windows, only the host file is listed.  So, if you’d embeded Notepad.exe in calc.exe, calc.exe would be in the task list even though you’d be running Notepad.  Nifty huh?
Of course, you’ll have to find a transport/archiver that supports these streams, if you want to distribute them.  Outlook and WinZip do not seem to do so.
Apparently ADS are there for backwards compatibility with a Macintosh file system (HFS), but I’m not sure why we’d want/need that…  Some Windows apps use it for other things.  For example when you download an exe file through IE, MS helpfully edits the file for you and attaches an ADS that stores the Zone under which the file was downloaded.  As a result you are NEVER getting the same file that was on the internet.  Developers should be aware of this when they deploy software via a website. 

In short, while Alternative Data Streams do add some flexibility to the operating system, the incomplete implementation of the feature makes ADS more of an avenue for abuse than a feature.

Steve Trefethen, Senior R&D Engineer Delphi/C#Builder for the Borland Software Corporation had an EXCELLENT post on his BLog. It emphasizes the cost of a mistake that I always see in example code on the web and in WordPress plug-ins as far as that goes.

Here’s the law that is always broken: “If you know an error is likely to occur in your code, check for that condition first.  DO NOT rely on exception handling detect it.”

This applies whether you are waiting for file access to fail  or an SQL insert statement to come back with an error.  Check to see if the condtion (or record) exists and then have your program handle it correctly.

Generic exception handling should only be used when you cannot predict or test for the condition. 

Newer versions (5+) of MySQL have a clause for inserts called ON DUPLICATE KEY.  Many MySQL programmers are starting to rely on it, but it too has a cost.  Even a single call that runs two insert queries with exception handling thrown in there is almost always going to be slower than checking for the record first and chosing what to do based upon the result of the read only select.  The ONLY reason I see for that call to exist is if it locks the database as it is running and therefore you KNOW the has not changed after your Select query is run. So it does have its application.

 Maybe this article will help others come around: 

BDS 2006, slow compile times and the cost of exceptions

While browsing the newsgroups I ran into a thread where a user had commented that BDS 2006 compiled his application incredibly slowly. A while ago I’d had a discussion with Mark Edington our resident performance maverick about this issue and it turns out that it’s a good reminder of the cost of raising an exception or in this case lots of them.

Since Mark debugged this issue he kindly provided me with a short write up of the specifc problem, it’s solution and a workaround.

function OpenOSFile(const Name: string): IStream;
begin
  { try opening existing file so that read/write operations are
    possible }
  Result := nil;
  if FileExists(Name) then
  begin
    try
      Result := TOSFileStream.Create(Name, fmOpenReadWrite or 
        fmShareDenyNone);
    except
      on EFOpenError do
        Result := TOSFileStream.Create(Name, fmOpenRead or 
          fmShareDenyWrite);
    end;
  end;
end;

Notice the try/except block, since there is no parameter to the function that specifies what access rights to use when opening the file, the procedure tries to be accommodating and provide read/write access first and if that fails it resorts to to opening the file as read-only. Unfortunately, as written, it introduces a serious performance penalty for anything other than small projects. Interestingly enough, the performance issue isn’t from the fact that it takes 2 attempts to get a read-only file opened, it is actually caused by the exception that is raised when the first (read/write) TOSFileStream.Create call fails.

In the test case that was submitted for diagnosing this problem the RaiseException procedure (which is a Windows API call), was accounting for nearly 40% of the total compile time. Under a profiler it took nearly 50 seconds to execute that procedure about 1100 times. That works out to around 45 milliseconds per call which may not sound like that much, but when executed 1100+ times it adds up.

The solution was very straightforward: Rewrite the routine and remove the exception based fallback algorithm. The new version looks like this:

function OpenOSFile(const Name: string): IStream;
var
  Code: Integer;
begin
  Result := nil;
  Code := GetFileAttributes(PChar(Name));
  if (Code <> -1) and (FILE_ATTRIBUTE_DIRECTORY and Code = 0) then
  begin
    if (FILE_ATTRIBUTE_READONLY and Code = 0) then
      Result := TOSFileStream.Create(Name, fmOpenReadWrite or
fmShareDenyNone)
    else
      Result := TOSFileStream.Create(Name, fmOpenRead or
fmShareDenyWrite);
  end;
end;

This version first checks to make sure the file is not marked as read-only before attempting to open it as read/write. The FileExists call that was replaced in the original version of the procedure is actually implemented using GetFileAttributes anyway, so there was no extra overhead introduced to make the check.

The moral of the story is avoid writing code that raises exceptions as part of the normal course of program execution. Exceptions are horrifically expensive to raise and handle and should be reserved for the truly “exceptional” events. If you have a medium to large Delphi project that has files marked as read-only the workaround for this issue is to mark them as read/write until this fix is made publicly available.

posted on Monday, February 27, 2006 3:51 PM