The tragedy in Microsoft's grammar checking software

In an email, I intended to say:

"it would be a matter of weeks before he could effect any changes.."

So Microsoft's grammar check wanted me to say:

"it would be a matter of weeks before he could affect any changes.."

(Yes, fair enough, I should have turned off the grammar check.  But I'd sucker myself if I spent all day tweaking every Microsoft annoyance.)


Oh so close but so far away.  Well not really.  They are different words, stored, analysed, parsed differently.  Rather, it suggests that the person putting together the algorithms succumbed to a simple high school error.


I acknowledge that it is valiant to even attempt grammar checking software.  But if you're game enough to release such, you must have some confidence in your product.  Or you're Microsoft, and your Quality Assurance has simply let another one slip past you.

SETI, Open source, and the socialisation of productivity

What does SETI have to do with Microsoft's furrowed brow?

We all know the Search for Extra-Terrestrial Intelligence, whereby the universe is scanned for signals throughout the electromagnetic spectrum which can be interpreted as originating with intelligent life. Some of us have run SETI@home: you download a screensaver, which runs in the background, borrowing your unused computer time to run a parcel of number crunching for SETI. Everybody wins: only your idle computer time is used, and it can have some wider community benefit - you may even be responsible for the first discovery of extraterrestrial life.

That was the first distributed grid computing project to gain widespread publicity. But the software is now available to turn any general project requiring major computer time into a socialised project. The Herald recently ran an article on Australian use of such software: specifically, BOINC, The Berkeley Open Infrastructure for Network Computing. The article said over 32,000 Australians were currently running BOINC projects, out of 1.7 million people worldwide.

The scope is tremendous, not just for general scientific research, but also for any community-sector project that may not otherwise have the resources to get off the ground.

For the moment, here's a list of projects you may wish to take part in. Those are all scientific research, mainly in biology, physics and maths, but there's also a World Community Grid, which is specifically aimed at humanitarian projects.

As for Microsoft, the other side of community computing is software: open source, to be specific: generally an open source project is contributed to by many, with no profit-oriented copyright - and generally available for free. Open Office may be the most famous - a direct competitors to Microsoft's Office suite. And as a method of developing software that is freely available to all, it has gained acceptance in most areas of my professional focus, business intelligence. Apart from the well-known mySQL database, there are also open source tools available for most related areas. As well as database and BI software, there's also ETL, data profiling, and so on.

Over time, you should expect prices to tumble in all types of software directly affected by open source initiatives. Yes, the likes of Microsoft can expect some buffering from these forces due to brand-name strength. But yes too, Microsoft is worried enough that they are already working on alternative revenue streams, including jumping into the cloud. Those alternatives shouldn't see a collapse of capitalism any time soon, but the long-term trend can only benefit the public, particularly those who might not otherwise be able to afford such computer resources, particularly in the developing world.

In a wider sense, distributed computing and open source are simply harbingers of a globalisation and socialisation of productivity, for the benefit of all.

Microsoft, installs and quality control

Some lessons in working with Microsoft - and in software installation - are over at my professional blog. Although there may be a familiar ring to the experience, I also have some thoughts on why it happens.

Windows: pirated, owned, bugged

Interesting to hear news that botnet malware is being found in pirated copies of the new Windows 7.

Here's the translation. Windows 7, not released yet, has been getting better press than the previous Windows Vista. Beta copies have been released by Microsoft, including the latest one available free from their website for the past few days. It's called "Windows 7 release candidate", and it's probably going to change very little for the proper release later this year.

But beta copies are pre-release, given to people for testing purposes: there could still be a few bugs in it. And W7rc is available from Microsoft's website, free. So why would anyone go for a pirate copy? In this case, they're being downloaded from a peer-to-peer, bit torrenting site. This means a much faster download: the bitstreams come from a number of sources, and so it's likely to be fast, not being dependent on a single server - or Microsoft's web site, for that matter.
Moreover, there'd be a number of people who are so used to downloading from such fast sources that they'd source their needs - licit or illicit - from there.

But some enterprising soul has hacked the W7rc code, just days after it was released. And they inserted into it code that compromises the computer it's loaded on, rendering it part of a botnet - a network of compromised computers that could be hijacked at will for any number of nefarious purposes, such as emailing spam or partaking in attacks on other computers (eg DDOS, distributed denial of service). And that hacker figured others in the shadow internet world would be sufficiently tempted.

And they were. Not only downloaded, but loaded, operational, and calling home to the specified target for orders. Dambala, an anti-botnet organisation "managed to grab control over" the server the hack was directed to, and noted that a peak of 550 infected computers per hour were calling in.

Lessons?

It should be one big bounty for Microsoft, in its quest to get everyone to pay them money: "But a genuine copy, or you'll get infected/compromised". This concept should also provide an even better bounty for evil hackers. Why stop at botnets? Why not a hack that allows for eavesdropping, so you can grab a user's personal information, hopefully bank accounts, etc.

Piracy will never be the same again.

BI Survey 8: business intelligence trends

Out now is the new edition of the BI Survey - once known as the OLAP Survey, now up to edition 8, covering 2008.

As a specific term, Business Intelligence is more widely known than Online Analytical Processing, but the Survey found that even on this, the software vendors are over-optimistic. They estimate about 14% of employees (of a "typical" organisation) use BI tools, but the Survey finds a reality closer to 8%.

Other points noted in the preview copy I have:
- BI implementations that follow a competitive evaluation of different vendor toolsets are more successful in every way - however, the Survey shows a slight drop in the number of competitive evaluations. My comment: this is not fully surprising, and I can give a couple of reasons: a) the competitive process is seen as too hard, costly, or lengthy; and b) encumbents with BI skills will inevitably veer towards the toolsets with which they are most familiar;

- the most common issue with BI tools is (query) performance - although vendors have a rosier view than business users. ;

- small software vendors provide better support than large ones.


The spate of vendor amalgamations is somewhat represented in the survey. Gone are explicit listings of tools such as ProClarity and Brio (subsequently Hyperion Intelligence), but still represented are TM1 (as Cognos TM1) and Hyperion (via Essbase - all their other tools were subsumed in the Oracle toolset).


Another blogger, Chris Webb, has seen the full report, and makes some comments here.

One interesting point he notes is about MicroSoft's marketing strategies vis-a-vis different product offering. He strongly suggests that MicroSoft is leveraging their BI tools (SQL Server services SSAS, SSIS and SSRS) to promote greater adoption of the MS Office suite - to the ultimate detriment of its BI tools. In effect, their BI toolset is less than could be in the interests of encouraging greater sales of MS Office. This is of note because MicroSoft are increasingly pushing their BI solutions out to Office products. Client-end tools in Excel were only the start.

The degradation of outcomes for one product in favour of another is certainly plausible for an organisation selling multiple products, especially if their marketing strategies are sufficiently sophisticated. However, I would in turn suggest that greater adoption of BI tools is hampered by specialist knowledge requirements, and that the more users are able to work within a familiar paradigm, the easier they find it to use the tools.

On the other hand, MicroSoft's BI solutions have always been far too heavily geared to the technical environment to the detriment of business-level stakeholders, and I'm not yet convinced their Office-related tools are a good fix.

Webb's other notable reading of BI Survey 8 is that MicroSoft's BI products come squarely in the middle of all rankings (eg usability, scalability, etc etc). This is not surprising. While they are capable of producing competent product, I don't think MS's BI products are stellar in the scheme of things - ubiquity is the word that springs closer to mind, since they come free with all enterprise edition databases. Thus MicroSoft will perpetually remain too big to ignore in the BI marketplace.

Marketing; Microsoft; the whiff of the nearly-new

Microsoft is talking up the virtues of their forthcoming release of Windows. In the process, the esteem of Vista is downgraded. Pundits, too, seem to feel freer to hurl brickbats at Vista.

Two of the common complaints are that it's far too memory hungry, and that it asks too many (security-related) permissions of the user. I'm certainly chagrined at the frequency with which it asks my permission twice to perform a single action. Doubtless there are particular reasons for each of those asks, but from a usability perspective, it reflects noticeably poor design - and/or integration.


The broad trajectory of Windows releases reminded me of a similar phenomenon. The general narrative runs as follows. A band would release a new album and pipe up in interviews that it was their "best yet". Pundits, too, would laud it. Yet upon release of a subsequent album, the previous effort would be written off with various excuses for why it didn't pass muster - both the musicians and the reviewers would indulge in this revisionism.

Marketing explains all this. Software publishers, record companies, bands, have a particular keenness to talk up their latest product. Once it is superceded, they have little further need to push the old product.

Of course, some bands have a little more integrity than that. But it remains that they're aiming to make a living from their published output. What excuses for the pundits and reviewers? In part, they may be too hasty in forming their opinions. Yet if they're not at least in part swayed by the material sweeteners of the marketers, then they have simply sold their judgments short.

SQL Server: varchar to real

I was having trouble loading some data into a SQL Server database (MS). The particular problem was converting a type char to type real.

There are several ways to change data types. Typically, you'd use cast or convert functions. But everything I tried returned the same message: "Error converting data type varchar to real."

I googled to find out how other people dealt with this, but found nothing useful.

Then I tried something that worked: convert the type varchar to type money, then to type real. Easy.

Of course, this makes perfect sense if your numbers only run to two decimal places. In fact, it works to four decimal places (the same if you choose type money or smallmoney, except that the latter is stored in fewer bytes).

The options for retaining precision past four decimal places are somewhat messier. One way is to multiply the numbers by a few factors of ten, but you still have to pay attention to the range of viable values for money or smallmoney.

I'm not convinced that this issue is yet another demonstration of the unwieldiness of MS SQL Server. All databases have their pain points; this is just another one.

Tech: Just what does IBM do?

It might surprise you to know that IBM is the largest I.T. company in the world - by revenue. Global sales in 2005: IBM: US$91 billion; Microsoft: about $40 billion.

Of course, it used to be the hardware giant: in the 1960s and 70s, IBM simply defined the mainframe computers that once formed the backbone of large enterprises. Then the mainframe market was slowly crippled over the 1980s and 90s by the rise of micro computers: they were effectively a victim of the success of... IBM-format PCs. And the Dells of this world have demonstrated that the PC sector is a dangerously low-margin market.

Over time, IBM turned to technical/business services and software. Some of that software is its own technology, like DB2 (the new release is discussed here: it includes native XML support and autonomic memory management).

But IBM has also been buying up software companies left right and centre, specifically to give it a full vertical business software offering.  In effect, it is seeking to lock in large enterprises by providing a full range of software/services across business needs.  This is a common trend in the larger software companies which is why, for example, Microsoft and Oracle have equally been on the takeover warpath for several years.


Like Apple, IBM deserves credit for successfully re-inventing themselves more than once. Whereas they once exploited their hegemonic dominance in the mainframe computer market to extract monopolistic profits (hence the epithet Incapacitating Business for Megabucks), they now operate much more competitively across a range of markets, leveraging off their brand name rather than their monolithic presence - something Microsoft is taking note of, as its own dominant position is being eroded by Linux, Open Office, and other open source offerings.


2009 update: Q2 2009 revenue reports at $23.6 billion - that's just the one quarter.  This is made up of:

• 57% services - made up of technical services (39%) and business services (18%)
• 22% software
• 17% hardware

The remainder is revenue from financing businesses to buy their upscale hardware.  More complete press reports on IBM's second quarter financial figures can be found here and here.

Tech: Gartner's Database market shares

Apropros to my comments about databasing everything, I note that the latest Gartner database report says the market (ie, relational DBMSs) has grown by 8% last year.

Is that all? Well, that’s the year-on-year increase in sales of new systems (as I read it), not a measure of the amount of information stored in databases.

However, of more interest is the market share. As you’d expect, Oracle’s far ahead, followed by IBM, with Microsoft only managing two thirds of IBM’s share. The actual proportions are 49% to Oracle, 22% to IBM, with Microsoft at 13% and Teradata only 3%. Surprisingly, Teradata’s market share has been static. Its tools were somewhat rudimentary when last I played with it, but it is quite an industrial-strength product for high-end data warehousing – quite a growth area. Probably indicative that IBM and Oracle are successful in maintaining their presence in that market segment.

What about open source? Growth is significant, but from quite a small base, so products like MySQL and Ingres are not quite on the radar yet.

Year-on-year movement in market share is also surprisingly static. There has been a slight increase from Microsoft, at the expense of most others, but the figures show they’re having more of a struggle than they bargained for, in trying to match the product – and brandnames – of Oracle and IBM. Still, those figures definitely count for something, as Microsoft’s entry point is considerable lower than all their direct competitors, so unit sales must have shown some oomph.


Of more relevance to my earlier comments, I guess, would be number of installations, amount of data held in each system, and year-on-year changes in those figures. If I hear anything useful, I’ll publish here.

Tech: Innovative IBM database and BI releases

Two recent initiatives from IBM are promising in terms of information integration and discovery.

Of course, since IBM lost to Microsoft its mantle as the most monolithic and pervasive entity in the I.T. world, it’s been working hard to re-invent itself. It’s even sold (to Lenovo, a Chinese company) its PC hardware business – the very business that fostered microcomputer standardisation, allowed Microsoft to gain pre-eminence, and ate away at its traditional, mainframe business. Their business is currently split between software, services, and mainframe hardware. Mainframes are now a niche market, and it’s their software innovation that garners attention.

On imminent release is Viper, software technology for their DB2 database platform which, amongst other things, allows for “native” XML databasing. The presentation I attended last week gave me the impression it permits admixtures of data with XML-defined data, but I’d be quite cautious about that until I could see it in action.

This is quite a dramatic initiative*, providing some enabling technology for the Semantic Web (discussed here and here). For me, the significance lies not simply in its ability to handle XML – which can be done in proof-of-concept by any number of vendors – but that it can do this natively, as an integral part of its DB2 product.

Also announced is IBM’s Content Discovery for Business Intelligence. Although this is a part of their WebSphere (application server) product range, in concept it permits pervasive business intelligence throughout an organisation’s structured and unstructured data. Provided, I presume, the unstructured data has been sufficiently tagged (manually or automatically). The announcement is careful not to include the term “data mining” so I’m a bit suspicious of its “discovery” nomenclature. Business Intelligence involves specific query, analysis, and reporting functions, whereas data mining is more a discovery of trends – the difference between asking specific questions and asking what patterns are in the data.

We’ll find out the full story when the dust settles. Still, access to unstructured data is nothing to be sneezed at. And if Viper can’t immediately database extant web pages, be sure that that’s the direction they’re going.

*1-June: In fact, it's been said to be not that dramatic, that Oracle has had native XML support for some time. I guess it's down to how genuine that "native" label is, and how they mix XML and non-XML. Comments welcome.
(Viper also adds range partitioning, which I can see being particularly useful in a data warehouse/business intelligence context.)

Tech: Business Intelligence: Hyperion and Microsoft seek shelter in each other’s arms

Unexpected news today that Hyperion and Microsoft are to integrate their Business Intelligence offerings.

Hmm. That report wasn’t greatly helpful: integration covers a multitude of sins. I had a closer look, via the respective companies’ websites. The press release is identical (except that Hyperion’s site was easier to navigate!); in fact, the announcement was made at Hyperion’s global Solutions 2006 conference.

The key word here is ‘interoperability’. They are planning to allow their respective products to work with each other. In particular, people will be able to access components of Microsoft’s SQL Server with Hyperion tools, and access Hyperion components from SQL Server Reporting Services.

Although the proof of the pudding etc, this is a significant initiative. In a number of ways, the two companies have become direct competitors. Like few others, Hyperion has built up a vertical solution package encompassing database, business intelligence, OLAP (online analytical processing), ETL (extract, transform, load) and a healthy range of tools relating to Business Performance Management. They’ve done this both by internal development and acquisition – with a particularly useful find in Brio for business intelligence [unfortunately, they’ve drowned the brand name, but you can find it as significant parts of Hyperion Intelligence v8, or Hyperion 9 BI+].

On the other hand, Microsoft is simply Microsoft. They mostly build their own (the SQL Server database technology was acquired from Sybase). As the elephant in the living room, they’ve diverged from other BI toolsets - as I’ve discussed before. Their delivery model tends to be “here’s the components in pieces on the floor. Oh, would you like one of our partners to do some consulting?”

However, by now they both have the full toolset and are going head to head.

Which is one of the reasons the news was unexpected – but good. Interoperability is good. It’s going to be essential for the survival of software companies, and it’s a boon for the information consumer. Hyperion and Microsoft each have something to gain from this initiative – although Hyperion more so, given the disparity in market presence. In fact, Hyperion has already integrated Microsoft's .NET development framework into a number of its products.

Hyperion seems to be doing what is needed to survive in its core areas. By comparison, a takeover of one BI company by another (as with Crystal Reports and Business Objects) is risible. Expect Hyperion and Microsoft to remain two of the main players in this space.

Tech: The future of Business Intelligence

In yet another Business Intelligence consolidation, Microsoft is buying ProClarity (see ITToolbox's report). ProClarity makes front-end, visualisation tools for use with Microsoft technology. This buyout makes sense, but it will no doubt scare other BI vendors - as does everything Microsoft does in their space, no doubt.

My feeling has been that the major BI products have been converging for some time, in terms of both look-and-feel and functionality. Apart from Microsoft, who has always had the muscle to go it alone. Yet this should scare the vendors more than most. Traditionally, Microsoft's solutions haven't really been solutions: they've been technology, which then needs to be implemented. Typically, Microsoft recommend one of their partners to actually design and implement a solution, which entails a fair bit of coding and tailoring. Sometimes a customer will have their own team of Microsoft programmers - with SQL Server 2005, .NET-based development plus the customer's inhouse database resources will normally suffice. However, the implementation will certainly take time and resources.

With the takeover of ProClarity, Microsoft will now have a better front-end - ie something more presentable for the customer, something that should take less time and resource to implement - albeit probably with increased training requirements.

I've said consolidation is good; it helps technology converge, and clears the field somewhat. I may be in the minority on this, as a Data Warehouse Institute poll has a majority of opinions saying it's bad for customers. But DMReview has a more sanguine view that simply points out the need to be aware of impending upheavals as mergers take hold.

For my money, we don't need a rash of competing technologies requiring evaluating customers to get their heads around a disparate range of tools. Ideally, we're moving towards more intuitive paradigms that don't require expert training, and convergence helps this. To date, though, convergence is going to snag on Microsoft vs everyone else. Indeed, we should expect more vertical consolidation (Microsoft+ProClarity, Hyperion+Brio) than horizontal (Business Objects+Crystal), simply because standalone BI vendors will need to seek shelter within an umbrella solution, rather than Business Intelligence in isolation. Why would customers really want the pain of selecting a database tool, then ETL, then BI, performance management, etc. At least Microsoft has that right.

I think the era of standalone BI vendors rising above niche markets is drawing to a close.