It turns out that, to Windows, UTF8 encoding is just another code page. Down in the RTL, the conversions to/from UnicodeString or WideString use the Windows API functions WideCharToMultiByte() and MultiByteToWideChar(). One of the parameters is the code page identifier to or from which the data is converted. If you pass CP_UTF8 (65001) to those functions, they’ll convert between UTF16 and UTF8. This is a lossless conversion. In Tiburón, we’re introducing an enhancement to declaring your own "typed" AnsiString. You have always been able to create a unique type based on any intrinsic type by declaring the new type with the "typed type" syntax:

type
  MyString = type AnsiString;

This would create a new type that is assignment compatible* with AnsiString, but with a unique type name and a unique RTTI structure. We’ve used this in VCL to distinguish normal "strings" from special strings such as TFileName or TCaption. By creating these unique string types, it was possible to create property editors that would be associated with a specific type of property, regardless of which component it used on or what the property name was. This is how only the Caption property on many components will automatically update the live design-time component as you type in the Caption value.

The thing is, with the above declaration, MyString will continue to always have an affinity for whatever the current runtime code page was. So, we introduced the following syntax for AnsiString only:

type
  MyString = type AnsiString(<1..65534>);

You can now control the code page affinity of any "typed type" AnsiString at compile time. The "parameter" to AnsiString must be a word constant expression. The values 0 and 65535 have special meanings. 0 is a normal AnsiString, and 65535 ($FFFF) means "no affinity." $FFFF is worth noting here as already being declared as a RawByteString. When assigning between AnsiStrings or passing them as parameters, if the code page affinity of the source and destination strings are different, an automatic conversion is done. In order to minimize potential dataloss during the conversion, all conversions go "through" UTF16. However, a string with an affinity of $FFFF tells both the compiler and the RTL, that none of these conversions should be done and to just move over the payload. In practice, however, there would be only a few instances of needing to use RawByteString, but it is there for your use.

So we now have the following declarations in the System unit:

type
  UTF8String = type AnsiString(CP_UTF8);
  RawByteString = type AnsiString($FFFF);

Like I stated previously, any assignment (or passing as a parameter to procedure or function) where the code page affinity between the source and destination are different, the payload will automatically be converted. Say you have a function that must only take UTF8 data. You can declare it like this:

procedure WriteUTF8Data(const Data: UTF8String);
begin
  // write UTF8 data to stream, file, socket, etc…
end;

Now, no matter what type of string you pass to this procedure, you know that the payload will have been coerced into UTF8. Pass a normal AnsiString, and the data arrives as the UTF8 version of that AnsiString converted from whatever the active codepage was or whatever that AnsiString’s affinity was set to. Pass a UnicodeString or WideString to the function and it too will be converted to UTF8. Pretty cool, no?

With the title to this post… All those physicists out there are going to hate me and Google.. hehe

*"Assignment compatible" means that you can assign or pass as a parameter from one "typed type" to another "typed type" or the intrinsic type on which it was based. They are not "var/out parameter" compatible. This means you can pass them by value to a function but not by reference.

Share This | Email this page to a friend

I just finished listening to a company (Embarcadero not Borland) wide conference call announcing to the whole company the closure of the Embarcadero+CodeGear deal. Wayne Williams, our new boss, made some very encouraging statements. Most notably was that just like ER/Studio, RapidSQL, and DBArtisan, Delphi/C++Builder are core Embarcadero product offerings. This means that these products are key to the business. Yes, there are many other products being sold, incubated and introduced, but it is the aforementioned products that form the pillars on which the company is based. Without them, many of the other products would not be possible. Like the foundation of a home, you just don’t take a metaphoric jackhammer to them and expect the structure (the company) to remain sound.

Another encouraging (or maybe scary, depending upon your perspective) point was that we are the last independent tools vendor with the breadth of offerings we have out there. This means no vendor or stack lock-in. We have tools for nearly every database and OS platform out there. We are also one the of the very few software companies that offer very strong non-Open Source tools right along side tools built either on or for Open Source stacks. JBuilder and 3rdRail are built on top of the very popular Eclipse framework. Delphi for PHP and 3rdRail are built for the very popular and widely used Open Source PHP and Ruby/Rails environments, respectively.

How things will change or even stay the same is still being planned and scoped. A lot of work had been done between the announcement of this deal and its close, but now is where most of the work can actually take place. Now that we’re no longer joined to Borland, we can now chart a new course under a new captain. I wish all the best for Borland as I’ve seen many happy and exciting days while there.

An interesting anomaly is that many of the CodeGear folks have had their service bridged, which means that some of us have, on paper, now worked for Embarcadero longer then they’ve existed :-). This is now day 6022 for me.

Share This | Email this page to a friend

For some perspective, I started when the Borland stock was around $80 a share (ouch). The new Borland Campus wasn’t completed. Borland occupied a reasonably large number of buildings throughout Scotts Valley. If Borland didn’t occupy some or all of a building, chances are Seagate did. To move between the various buildings, there was a continuously running shuttle bus service for Borland employees. I also remember the clout that Borland had locally. When we (my family and myself) first arrived and were looking for someplace to rent, when asked where I worked and I said Borland, suddenly we were at the top of the list without any kind of other checks. It was pretty nice.

After today, there will be no official presence of Borland in Scotts Valley. The only thing linked to Borland will be name on the lease they maintain on part of this building that was once the Borland Corporate HQ and campus. Embarcadero will sub-lease the space we currently occupy from Borland.

Even though there have been many things Borland has done over the years that I firmly disagreed with, they’ve also done some pretty good stuff too. I’ve learned so much more than I ever could have imagined about software, the software industry, developer tools, frameworks, and compilers. I’ve also had the pleasure of knowing and meeting many folks in the industry from all around the world.

Tomorrow will start a new chapter with new challenges and opportunities. I’m sure it will take some time until the dust settles, the cultures merge, and things settle down to the right groove. Don’t expect any earth-shattering announcements or radical changes immediately out of the gate. This is a new experience for nearly all involved, so some period of acclimation is bound to take place.

Share This | Email this page to a friend

Throughout the day, yesterday, I got into several IM conversations from former co-workers all asking how I felt about the news. I honestly told them that this is the best outcome for the future of the developer tools products anyone of us here could have imagined. As I was on my way from work last night, I got a phone call on my mobile phone. I was wondering when I would get this call. The voice on the other end said, "Hi Allen, this is Chuck." Yep, I was totally expecting a direct phone call sometime from Chuck. If you’ve been following any of my posts over the last few years, you know that Chuck Jazdzewski was very instrumental in my getting hired at Borland. Chuck served as my mentor for many years.

Chuck’s first question was one of concern and wanting to know how I felt about these developments. Thanks, Chuck, that meant a lot. We talked for a while as I was driving. Don’t worry, I use a bluetooth hands free headset :-). Part of the conversation quickly drifted to our many conversations in the past about how cool it would be to take the dev tools out of Borland. Unfortunately, I arrived at my destination and had to cut the conversation short. I would like to have continued. Chuck agreed that if he had any other questions that he’d be sure to give me a call.

If there is one thing about my tenure with the Borland developer tools group, it has rarely ever been a dull moment. I doubt that is going to be changing any time soon. Change is something the CodeGear teams have become accustomed to. It’s great that this change is one of the most positive ones :-).

Share This | Email this page to a friend

At 9am PDT, we held a company "all-hands" meeting here in Scotts Valley, CA. Since this meeting coincided with Borland’s earnings announcement, this was not an unusual event. Oh sure, the internal rumors had been flying for months. Folks kept saying that "something is happening." After the usual presentation and outline of the deal, there was an opportunity for a Q&A session with Jim Douglas, the CodeGear CEO. Jim was candid and frank. Like anything that is a major change, there is bound to be some level of nervousness. In this case, the employees were far more engaged in the meeting that I’ve seen in the past. It seemed that there was this collective sigh of relief among the whole crowd. We finally made it!

Michael Swindell presented information about how this is a great fit for both companies and began outlining a lot of the immediate and long term opportunities. When something fits this well, it’s easy for people to just "get it." It’s not a tough thing to "sell." If you’re a CodeGear customer, I would encourage you to head on over to Embarcadero’s site and just browse around and look at what they’re offering. You should also take a moment to read this blog post from Greg Keller.

Throughout most of today, intermixed with doing a little bit of coding, I’ve also been cruising the newsgroups, blogosphere, and other message boards. So far the overall tone is very positive and upbeat. There are lots of questions and concerns. It’s only the first day, so I’m sure as folks have a chance to digest what this all means, the real questions will start flowing. Off the top, right now there are no plans to veer from our current roadmaps or to interrupt any current schedules. As a matter of fact, this was the directive given from the Embarcadero folks to make sure we’re still on track to deliver what we’ve committed to.

I also spent some time roaming through the halls here in Scotts Valley.  After the 9am meeting, there was a lot of hallway conversations about what this all means. There was no "hand-wringing" and "angst" among the team that I could see. In fact, most folks were already talking about what Embarcadero does, the products they offer, and how they fit with what we’re doing. Those are exactly the conversations we should be having internally. By the afternoon, most of the team had settled back into their routine and the development engine was back running on all cylinders.

So if you have any questions or comments, feel free to let me know and post through the comment system here. As I already stated above, it is business as usual in terms of products, schedules and features. As we approach the final close of this deal, more and more information will become available. So if I cannot answer your question right away, I hope to be able to do so in the coming weeks.

UPDATE: It seems that many of you are already hitting Embarcadero’s site.  It appears to be suffering from a mild case of "slashdot effect". This is a good thing. Let’s make sure they know how engaged, passionate, and committed the CodeGear customers are.

Share This | Email this page to a friend

Well? Is that not some interesting news? If you’ve been wondering why this blog has been somewhat quiet lately, it’s been due not only to being "heads-down" on Tiburón, but also because of this news. It’s been a long-time in coming, that’s for certain. In the coming days, you’re sure to hear a lot more about what this means and comments from various sources.  Once I’ve had a chance to gather together some more information, I’ll be sure to post it here.

Share This | Email this page to a friend

I’ll get to the real motivation for swallowing my pride and buying the Mac in a moment. This article popped up on my RSS feed and it really resonated with me.  Regardless of the comments, which all too painfully only served to prove the author’s point (and which I’m sure there will be comments to this post which will do the same thing), it was interesting to see something I’ve long suspected, actually put into words.  It was also so apropos since I’ve, sort of, joined the ranks of Mac owners. The gist of this article and the book from which it is excerpted (from what I can gather), is that we’re all very passionate, like to "belong," and want to feel validated and sometimes even lauded for our choice in car, home, clothes, spouse, occupation, religion, computer, etc… Yes, I admit that I sometimes feel much the same way as the stereotypical Apple-fanboy when confronted with valid, well-reasoned, unbiased analysis.

This hit home all too well lately because, well, this new Mac is not really mine.  Yes, I was involved with its purchase, but I don’t actually use it.  See, my wife has decided to return to school and formally pursue something she’s come to really enjoy over the past few years, graphic design and layout.  It started with doing monthly newsletters for the local Mothers of Twins club (my daughters are twins), then on to doing other various projects, and finally culminated in doing the entire Scotts Valley High School annual Football program.  She’s already committed to do it again for the Fall-2008 season. She has used PCs (natch) for many years and has a very nice 17" notebook. Now that she’s back in school, which is where she’s at while I write this post, it became painfully obvious that there is a very clear bias in this particular field toward the mac.  Sure, the same applications are available for the PC, but to be truly "accepted" shouldering a mac really helps. Most printers happily work with raw files from these specific applications and will actually charge hefty fee if you submit work using certain PC-only applications. It also makes the professor actually want to talk with you.

Here is where this whole Mac-fanboy, "Apple can do no harm," attitude really hit me.  As relayed by my wife, this professor unequivocally stated on the first day of class that you must use the Mac version of the applications.  The pure shock and horror on the professor’s face when asked if one could use the PC versions had to be priceless. A similar attitude was expressed when my wife submitted some homework which had a few things marked off.  The professor actually told her that she must not have used a laser printer because a laser printer would have printed correctly.  Well, that chapped my hide to no end… I wanted to load up my nice COLOR laser printer into the truck, drag it up to her class and plop it into the middle of the professor’s classroom and insist that they prove to me that this is not a stinkin’ laser printer! (no, it is not one of those dye-sublimation or "melted crayon" printers) That lasted for only a few moments as I stood there speechless and dumbfounded.  Upon further examination and obtaining more information, it turns out that it wasn’t printed on the same brand and model laser printer that the professor used.  See this is a design and layout class for printed medium, and to grade the work, it is not about the content (which is usually that funky fake Latin text), but about its layout and presentation on the page.  So where things are is more important that whether or not you spelled some word correctly.  Ok, that makes sense, she’ll have to use the school’s laser printers instead of ours.

The whole purchasing experience was just too surreal.  You walk into an Apple store, and it is literally like being a voyeur to this whole cult-like Apple… thing…  I just cannot explain it.  We’d already researched exactly which Mac we’re going to get and we’ll use my wife’s student ID for a discount.  It was simply a matter of walking up to the machine we wanted, point to it and grunt a few syllables about how we’re going to pay for it and instruct the resident drone to walk back through the stainless steel door at the back of the store to obtain said box of Jobs’ magic.  The top-of-the-line 17" MacBook Pro is certainly a sexy bit of hardware. You just cannot deny that. That was nearly four weeks ago. At about two weeks into this, things started going a little south on us.

Once some of the applications needed for the class arrived (after getting some nice deep student discounts), it was time to install them.  I get this frantic message from my wife saying that the machine won’t recognize the disk, and now it won’t even boot!  Hmmm, ok…  not. good.  Oh, yeah, the whole "see I knew it!" flags started popping up in my head all over the place.  They do have flaws!  Eventually, she was able to eject the disk, after making some really horrendous noises and still never recognizing the disk.  Less than two weeks old, and the Super-drive is dead.

You gotta love this, you can make an appointment at the "Genius bar" online.  So we did.  Drove back to the store and met with the "Genius" (yes, I’m snickering here…).  We explained that we’ve had the machine for barely two weeks and only now had an occasion to use the drive and this is what happened.   They tested it and, sure enough, no workee.  Just nasty noises.  I was ready for what happened next.  I asked if they’re going to "send it in" for repair.  I was all ready to explain that if they went down that road, I’d be a little irritated because, to me this was a failure that left the factory in that condition and why should we be without the machine for however long it takes to fix it.  Also, I’d just be inclined to return it for a refund.  Luckily, they saw the logic agreed to simply replace the machine.

Yeah!  They’re going to replace it… not so fast, there buckaroo. It turns out that we’d bought the machine only a few days before Apple decided to announce some minor changes to the whole MacBook line.  New things like LED backlit LCD, multi-touch track pad, newer Penryn-based CPU.  That all meant they were out of stock.  Rats.  They said they’d call once the new machines were in.  (I’ve heard that song and dance all too often) For the next week, we called the store several times and still no shipments. We contemplated simply getting a refund and ordering online, but that would mean not having the machine for school.  Finally, yesterday, they actually called. A new machine had been set aside and to come in anytime.  After arriving, we explained why we’re there and they walked back through those sterile stainless steel doors, and returned with a box to which was affixed a post-it note with my wife’s name on it.  Nice.

After about another hour as they transferred the documents, settings and applications, we were on our way.  And, yes, I did have them test the drive to make sure it worked :-).  All-in-all, Apple’s service was pleasant and understanding.  I’d have to say that the hardware is no higher or lower quality than a solid ThinkPad or some of the newer Dells (older Dell notebooks had, ahem, some real issues).  They’re certainly more "artsy" looking than a pure utilitarian PC notebook. And OS X? It’s just different. Some things I like about it, and others just make me want to scream. I can say the same things about XP or Vista.

I’m sure many will see through my "unbiased" view straight to the obvious bias in this post.  And for those Apple-haters, I hope this redeems me; sitting next to me here in my home office, is a new, home-built, overclocked quad-core, 4GB, Vista x64, 512MB nVidia 8800, 1TB raid, PC rig that I had to build to seek forgiveness and cleansing from committing  Apple-adultery ;-).

Share This | Email this page to a friend

When I started down this road, it all began as an investigation into various ways of implementing a thread pool.  It turns out that this simplistic mechanism really doesn’t truly address parallelism and concurrent processing.  It is merely a way of shoveling off a task into another thread, which may or may not execute concurrently.  A thread pool doesn’t do anything to "load-balance" the work in a constructive manner, nor does it purport to.  That is left as an exercise for the user.

While reading and researching all the various approaches to handling parallelism, a few light bulbs started coming on.  I’ve been approaching this whole thing from the wrong angle.  I’d been trying to shoehorn task handling (a task being the smallest unit of work that can be made parallel) concepts into the more general notion of a thread pool.  I admit that sometimes I’m not the sharpest knife in the drawer, but I can recognize when one approach is superior to another.

Rather than starting with the concept of thread pools, a parallel library should really begin at the most basic level, the task or just a unit of work.  A task is merely some chunk of code that may or may not execute concurrently.  Another thing to remember is that a parallel library is about getting these tasks done as fast as absolutely possible.  It is also about keeping as many of the CPU cores as busy as possible with as little idle time and overhead.  It is not about fairness. It is about speed.

To best do this, the internal scheduling and management of tasks should control how they are scheduled and processed on the CPU cores.  Remember, we’re talking about the actual real, hardware CPU core (or virtual core, like Hyperthreading).  Not a thread.  A thread is merely a way to easily segregate tasks among the cores.  We’ll rely on the operating system to make sure that of the available cores, then there are equally as many "threads" actually burning CPU cycles for real work.  Any other threads are stuck waiting for their chance at a core.  Because of this, a task manager should endeavor to keep the number of running threads as close to equal to the number of available cores.  It should also equally handle 1 or 100 cores by making sure as many cores are a busy as possible.

Using a task based approach also requires that the threading and scheduling is completely (or nearly completely) hidden.  The user should never think of threads.  The user only deals with Tasks and the extra abstractions built on top of them.  So it looks like DPL is headed for another rewrite, which is good because each time I do this it is only getting better. I’ve not spent a whole lot of time on the current iteration as it stands, so this isn’t much of a setback. 

Microsoft’s TPL is one of the few things to come out in recent years that looks great and performs well.  The concepts behind it are very well suited for Delphi.  To get an idea of some of the things you can do with it (and eventually the DPL you really should watch this video.  The really interesting bits come on toward the last 1/2 to 1/3 of the video where they go into the idea behind work stealing and work load balancing.  Really, really interesting stuff.  Joe Duffy also addresses how you need to think about parallelism and how it applies to your situation.

As part of this, I just ordered Joe’s soon to be released book on Concurrent Programming on Windows Vista.  You can actually buy the book online ahead of the actual print release and download a PDF of the work in progress until then.  From reading only a few of the online excerpts, it looks to be one of those "must have" tomes for anyone doing any kind of parallel programming in native Windows and in .NET.  Even though a lot of the information I know, it is good to have it codified in a consistent organized manner.  As I stated in my first post about thread pools, as I dig deeper into this whole subject, I want to make sure that Delphi developers at all levels will find some value in such a library.  The best thing you can do at this point is to learn as much about these concepts as you can.  Hopefully this blog and others I’ve been referencing and will reference in the future will be an excellent source of information.  I’m finding that as I write about this, it give me a chance to review my assumptions and design ideas with a critical eye.  Hopefully, we can all learn something along the way as well.  I know I have so far.

Share This | Email this page to a friend

A Critical[Section] Difference: Windows XP vs. Windows Vista
Breaking the rules
Simmering Unicode, bring DPL to a boil (Part 2)
Simmering Unicode, bring DPL to a boil
Placing your code in the forge - Refining a technique
When code lies - A better solution
Stupid Enumerator Tricks - And now for something completely different
Magical Assembler Incantations - Nested functions and anonymous methods
The Life and Times of a Thread Pool
I cut and I cut and it was still too short!
Spot the deadlock
Wading in the shallow end of the pool - Thread Pools

As another follow on to my discussion of the TMonitor (see the above "Bring DPL to a boil" articles) things have changed a little. The TMonitor class is no longer a separate class that you can create and use. It is now tied directly to any TObject instance or derivative (which means any instance of a Delphi class).  There is still the notion of a TMonitor but only to serve as a way to tie together the "monitor" functionality. Rather than creating one and setting about to locking/unlocking it, you now only need an object instance. For Tiburón, you merely need to call System.TMonitor.Enter(<obj>); or System.TMonitor.Exit(<obj>); among the other related methods.

I’ve contemplated putting these methods directly on TObject itself, but there is the issue with calling these methods when a descendant class has a method of the same name. The code will compile and function normally due to Delphi’s scoping rules, but it could make it harder for users to call those methods in these cases. Thread synchronization should never be done lightly and should require some forethought and planning. A simple rule of thumb is that you should never call unknown code (or code that you’re not entirely sure where it goes and what it does) while holding a lock. 

This now paves the way to add some interesting intrinsic functionality such as using this as the basis for "synchronized" methods or even synchronized code blocks similar to the C# lock keyword. While you’ve always been able to create an OS critical section, or use some of the helper classes in SyncObjs.pas, this new mechanism will be available on any object.

I merely design it then write about it… you get to kvetch

Share This | Email this page to a friend

LeaveCriticalSection(CSec);
EnterCriticalSection(CSec);

Yes this is wrong on so many levels, but the interesting thing to note is that under Windows XP, the above code hangs at the EnterCriticalSection call because the LeaveCriticalSection call did some very bad damage to the critical section structure.  The problem is that the RecursionCount field of the critical section is decremented and left as non-zero (-1) which causes the LockCount field to also be decremented in order to keep the two field counts in sync.  When the EnterCriticalSection call is made, the LockCount field is incremented, but it is still non-zero which makes it think there is another owner.  The OwningThread Field is compared with the calling thread’s ID which being 0, is not equal to the calling thread.  But no other thread owns the lock!  It blithely forges ahead and allocates the wait event and proceeds to block.  Oops!

Windows Vista, in contrast, has changed how it manages the LockCount field.  Rather than only accumulating the waiters (and recursions) in terms of simply incrementing (or decremented) the LockCount field, it uses the low-bit as the actual indication of holding the lock and accumulates the waiter and recursion counts in the remaining bits.  This means that the critical section can now only have 2^31 recursions and waiters. (UPDATE: Upon further examination, only the waiters are indicated in the LockCount field)  Still plenty of space.  The upshot of this is that the above degenerate case no longer will cause a complete hang in many cases where it would have.  Why they made the change is anybody’s guess;  Maybe Raymond Chen can pipe in and explain the reasons…  Seems to me that anyone with doing the above deserves to be put into the penalty box.  Now all that is going to happen is that when a program that once used to hang, will now appear to work only to probably show up some other flaw downstream.  Seems like a hang/crash now or hang/crash later kind of thing.  The end result is still the same.

Share This | Email this page to a friend