Here is the answer, from stackoverflow:
I have been fighting this one as well, and have just found a solution that actually makes sense. I ended up having to build a debug version of the WindowsAccessBridge.dll and used the debugger to step into it to watch what was happening. ?The call to 'initializeAccessBridge' REQUIRES you to have an active windows message pump. Inside 'initializeAccessBridge', it (eventually) creates a hidden dialog window (using CreateDialog). Once the dialog is created, it performs a PostMessage with a registered message. The JavaVM side of the access bridge responds to this message, and posts back another message to the dialog that was created (it appears to be a 'hello' type handshake between your app and the java VM). As such, if your application doesn't have an active message pump, the return message from the JavaVM never gets received by your app.
This is important as until this message is received, the bridge is never properly initialized and as such all calls to 'IsJavaWindow' fail (internally, the bridge initializes an internal structure once the message is received -- as such, no active message pump, no initializing). I'm guessing that this is why you never receive callback messages as well.
Not only that, but you must call initializeAccessBridge at a point where the message pump can process messages before you can call IsJavaWindow.
This is why JavaFerret and JavaMonkey work -- they initialize at startup, and then enumerate on response to a menu message, well after the bridge has received the initialization message via the message pump.
The way I was able to solve this in my MFC dialog app (and our MFC-based application), is to make sure that you call 'initializeAccessBridge' at a point such that the built-in MFC message pump can push the 'hello' message back to this hidden dialog BEFORE you use it. In the simple MFC dialog case, it meant calling initializeAccessBridge in OnInitDialog, and calling the enum procedure in response to a button call (for example). If you want the enum to occur as soon as the dialog appears, you could use a timer to fire (eg. 10ms) after the OnInitDialog completes to allow processing of the initialization message.
If you are planning to use this in a console app, you will need to write your own custom message pump to handle the initialization message.
Anyway, I hope this is clear enough! Whilst there is no way to know whether this is the 'correct' way (other than to pay for a Sun engineer to tell us), it definitely solved my problem.
Cheers -- Darren.
oh. and btw I found an obscure Sun page that mentioned something about the AccessBridge only working for awt-based java apps (but given that Sun hasn't updated any documentation since 2004 this may have changed). I'm not a java programmer -- for testing I grabbed a number of free Java applications (as well as the ones that came with the jdk) and tried out my test application. It worked for all of the ones I tried -- YMMV. Good luck!