If you don't mind working in Objective-C, read this:
http://developer.apple.com/Mac/library/documentation/Cocoa/Reference/ApplicationKit/Classes/NSWindow_Class/Reference/Reference.html#//apple_ref/occ/instm/NSWindow/setStyleMask:
You'll have to copy/paste that link, since the [a ...] apparently can't handle URLs...
If you want to do this entirely in BlitzMax, prepare for a world of hurt.
Step 1: get this: http://github.com/nilium/pub.objectivec
Step 2: read this: http://developer.apple.com/mac/library/documentation/Cocoa/Reference/ObjCRuntimeRef/Reference/reference.html
Step 3: use steps 1 and 2 to accomplish the calling of NSWindow -setStyleMask:.
E.g., like so:
Import Pub.ObjectiveC
...
Local window:Byte Ptr = <the window>
Local _windowStyle%(o@Ptr, s@Ptr)=Null
Local _setWindowStyle(o@Ptr, s@Ptr, style%)=Null
Local clas@Ptr = object_getClass(window)
Local setsel@Ptr = sel_registerName("setStyleMask:")
Local getsel@Ptr = sel_registerName("styleMask")
_setWindowStyle = class_getMethodImplementation(clas, setsel)
_windowStyle = class_getMethodImplementation(clas, getsel)
_setWindowStyle(window, setsel, _windowStyle(window, getsel)|(1 Shl 2))
...
| 
