Development Activities and Methodology Usage by Australian Web Site Consultants ABSTRACT

Site Consultants

John R. Venable and Fion C. B. Lim School of Information Systems

Curtin University of Technology, Australia E-mail: venablej@cbs.curtin.edu.au

ABSTRACT

Consulting organisations that develop web information systems (WIS) were surveyed by postal survey to determine their development activities and their usage of formal WIS development methodologies (WISDMs) as well as methodologies embedded in web development tools. The results showed that fewer than 10% of survey respondents use a WISDM to guide their WIS development activities. Fewer than 15% use any non-tool-embedded methodology. Only 67% use a methodology of any kind, embedded or not. Indeed, many WIS developers were not even aware of methodologies specifically for WIS development. The results also showed that the emphasis during WIS development was on design and implementation activities rather than earlier analysis activities.

Keywords: development methodology, web information system, web engineering, systems analysis and design

INTRODUCTION

Studies of web sites and web information systems (WIS) and their development have shown that WIS are of poor quality and expensive to develop and maintain (Lowe and Webby, 1998). According to a report from the first International Conference on Software Engineering Workshop on Web Engineering (Jones, 1999), ad hoc development methods have cause a crisis in web development. Web information systems are information systems (IS) that are delivered over the Internet. Isakowitz et al. (1998) defined WIS as “systems that organizations and their clients use to conduct electronic commerce.” For our purposes here, we distinguish WIS from simple web pages that only present static information. A WIS may be embedded within a web site, which may also include static web pages. In the IS field, the use of an IS development (ISD) methodology has long been postulated as a means of improving ISD. Postulated benefits include reduced cost and time in ISD, improved quality of the resulting IS, and improved benefit realisation by the organisation that uses the resulting IS.

While we agree with Stamper’s (1988) concern that the term methodology is misused, we similarly “bow to customary usage.” A methodology can be defined as … “an explicit way of structuring one’s think and actions. … A methodology will tell you ‘what’ steps to take and ‘how’ to perform those steps but most importantly the reasons ‘why’ those steps should be taken, in that particular order.” (Jayaratna, 1994, p. 37) Many ISD methodologies have been proposed and used and the area has been heavily researched, e.g. in the conferences of IFIP working group 8.1.

Given that WIS are a kind of IS, and that ISD methodologies can be used to improve ISD and the resulting IS, the questions that interest us here are:

• To what extent are ISD methodologies used to improve the development of WIS? • In what phases of WIS development do practitioners engage?

• What levels of emphasis do practitioners place on the different phases of WIS development? Our research on these questions has focused on the Australian context. Web Information Systems Development Methodologies

A Web Information System Development Methodology (WISDM) is an ISD methodology that is specifically aimed at development of a WIS. A number of WISDMs have been proposed. Bauer (1998) classified WISDMs as being of three different categories: (1) those using modified models

4th Western Australian Workshop on Information Systems Research (WAWISR 2001)

John R Venable & Fion C B Lim

based on traditional IS or software development methodologies, (2) those based on hypertext modelling, and (3) those built into web site development tools. We give a brief overview of those WISDMs for which we surveyed.

The Relationship Management Methodology (RMM, Isakowitz, 1997) falls into category (1) above as it builds on and adapts the ER model. RMM provides a seven-step process, including:

1. Information architecture

2. User interface and navigation design 3. Content creation and authoring

4. Workflow and document management 5. Publishing

6. Document review and link management 7. Search and retrieval RMM utilises a CASE tool as well, called the Relationship Management CASE (RMC) tool. RMM has been criticised as only being applicable to a limited domain of web sites, which emphasise database interfaces, because of its poor handling of spatial elements (Lima and Eng, 1998) and of the possibilities for rich information presentation on web pages (Isakowitz et al., 1997).

The Object-Oriented Hypermedia Design Model (OOHDM, Schwabe and Rossi, 1995) falls into category (2) above as it is based on a hypermedia model. The methodology has a four-step incremental prototyping process in which object-oriented models are built of four design concerns. The steps are:

1. Conceptual design (classes, relationships, attributes, and perspectives) 2. Navigational design (nodes and links)

3. Abstract interface design (interface classes, e.g. text fields or buttons) 4. Implementation (mapping interface objects to implementation objects) OOHDM has been criticised because data relationships may not reflect the users’ needs (Lee et al., 1999) and because there is poor support for linking to back-end databases (Takahashi and Liang, 1997).

The Dexter Hypertext Reference Model (Halasz and Schwartz, 1994) also falls into category (2) above. It models a WIS in three layers: a storage layer, a run-time layer, and a within-component layer. The storage layer focuses on a database of nodes and links. The within-component layer is the content of the nodes. The run-time layer handles the dynamics of the presentation of the nodes and the user interaction.

Grønbæk and Trigg (1996) have criticised the Dexter Hypertext Reference Model because the explicit separation of links from nodes does not deal well with the embedded nature of links within the content of nodes.

The Natural Language Generation (NLG, Lima and Eng, 1998) approach emphasises the information storage and retrieval aspect of WIS and the need to move quickly from one document to another. As such, it is also in category (2) above. The NLG approach has three tasks:

1. Context selection (deciding what should be communicated) 2. Text structuring (organising the information for presentation)

3. Text planning (composition of a discourse plan based on facts in a knowledge base) The NLG approach has been criticised as not dealing practically with the way users of the WWW read documents, i.e. they skim read rather than read in detail (Dale et al., 1998). This reduces the benefit of carefully organising pages to eliminate redundancies.

The World Wide Web Design Technique (W3DT, Bichler and Nusser, 1996) methodology was developed as a more comprehensive way of handling both the hypertextual richness and complex information content of WIS. In supporting both static and dynamic components as well as sites whether they are relatively well structured or unstructured, it attempts to cover both categories (1) and (2) above. In the methodology, a number of diagrams are developed which model the WIS hierarchically, deferring detail to lower level diagrams.

2

Development Activities and Methodology Usage by Australian Web Site Consultants

In addition to the formal methodologies above, Commercial Software methodologies are those in category (3) above. They are implied/embedded in many web development tools, such as Frontpage, Dreamweaver or Netfusion. These methodologies tend to be minimal and focused on design and implementation. SURVEY METHOD

A postal survey was developed to answer the above questions (in addition to other questions). With respect to each of the above methodologies, survey recipients were asked whether they were:

• Currently using the methodology • Had tried it more than once • Had tried it once

• Had never used it before • Had never heard of it before. Recipients were also asked the same question for which they could supply any other methodology not identified above.

Additionally, survey recipients were asked what percentage of time was spent on each of the following phases:

• Planning (team formation and scheduling)

• Analysis (business and WIS objectives, business analysis and systems analysis) • Design (visual/view design and information/navigation design) • Implementation (programming and scripting) • Testing

• Documentation • Promotion

• Measurement (and analysis with surveys or log files, etc.)

• Post Launch Activities (continuous analysis, re-development, new functions) • Other (to be specified by the respondent) The sample of web site developers to be surveyed was developed by searching the net for “web site consultant” (within Australia) and by referring to web sites that could provide that information. Potential recipients were checked for appropriateness by reviewing their home pages for suitability. A total of 278 questionnaires were posted, of which 236 reached a valid member of the desired population (others being returned unopened, not contactable on follow-up at that address/telephone, or later identified as not being appropriate). Of the 236, despite follow-up telephone calls, only 22 responses were received from people qualified to answer the questions. The response rate of 9.32% was disappointing (10-50% being normal for postal surveys according to Neuman, 1997). Non-response bias is discussed in the discussion section further below. The findings are detailed in the next section. FINDINGS Demographics

On average, the organisations that responded were fairly new and quite small. The average number of years that the organisation had been in the industry was 2.73 years. The average number of web development staff was 4.31. The average length of a typical project was 8.27 weeks. The average cost of the organisations’ smallest project was (in AUD) $2,271 while the average cost of the largest project was $34,090.

The respondents characterised their business as follows: 77% develop business-to-business (B2B) WIS, 77% develop business-to-consumer (B2C) WIS that are simple marketing tools, 82% develop B2C WIS that support customer interaction, and 55% develop intranet (i.e. intra-organisational) WIS.

3

John R Venable & Fion C B Lim

WISDM Usage

A very low percentage of the respondents (see note below) use any of the five WIS development methodologies for which we surveyed. Percentages for each of the different WISDM are shown in Table 1. Overall, only 23.8% had even tried any of the five WISDMs. In all five cases, over 50% admitted never having heard of the WISDM and 38.1% had never heard of any of the WISDMs for which we surveyed. Of the WISDMs for which we surveyed, RMM and W3DT were the only ones tried and used. Only 9.5% were currently using a WISDM and only 14.3% were currently using any methodology that wasn’t embedded in the software. Instead 61.9% of the respondents were using methodologies embedded in tools. Overall, 66.7% were using some form of methodology to guide their WIS development, while 33.3% were not using any methodology.

* See note below

RMM OOHDM Dexter NLG

Hypertext Approach Reference Model

W3DT Web Other

Development

Tool Methodology

61.9% 4.8% 4.8% 0.0% 4.8% 0.0% 9.5% 4.8% 19.0% 90.5% Currently

use Tried more than once Tried once Never tried Never heard of

4.8% 0.0% 0.0% 0.0% 4.8% 0.0% 0.0% 0.0% 0.0% 9.5% 9.5% 0.0% 0.0% 0.0% 0.0% 33.3% 47.6% 23.8% 33.3% 28.6% 52.4% 52.4% 76.2% 66.7% 57.1% Table 1: Web Consultant use and familiarity with WIS Development Methodologies

*Note: Figures in Table 1 are based on 21 responses. We threw out one respondent who indicated that

they were currently using all of these methodologies, including “other”. The organisation in question had only two developers and we consider that this circumstance is impossible.

Instead, most web consulting organisations use methodologies that are embedded in web development tools. 61.9% reported that they were currently using such a methodology. However, overall a full 33.3% reported that they were not currently using any methodology, not even another, in-house methodology.

Development Phase Emphasis

Table 2 and 3 show the percentages of project time spent on each phase/activity. Table 2 is sorted by the nominal chronological order while table 3 is sorted by percentage of project time.

Phase/Activity Percentage Phase/Activity Percentage Planning 6.66% Implementation 31.67% Analysis 16.39% Design 23.28% Design 23.28% Analysis 16.39% Implementation 31.67% Testing 7.36% Testing 7.36% Planning 6.66% Documentation 2.85% Post Launch 5.75% Promotion 2.31% Documentation 2.85% Measurement 2.82% Measurement 2.82% Post Launch 5.75% Promotion 2.31% Other 0.92% Other 0.92% Table 2: Chronological phase percentages

Table 3: Phase percentages by percentage

Additionally, the more detailed activities within Analysis, Design, Measurement, and Post Launch areas were investigated.

4

Development Activities and Methodology Usage by Australian Web Site Consultants

Of the 16.39% reported as devoted to analysis, on average, analysis of business and web site objectives constituted 6.30% of project time. Business analysis also constituted 6.30%. Systems Analysis constituted only 3.79% of project time.

Of the 23.28% devoted to design, visual/view design constituted 15.64% of the project time while Information/navigation design constituted 7.64%.

Of the 2.82% devoted to measurement, 1.10% was for direct feedback, such as by online survey while 1.72% was for indirect feedback, such as by analysis of log files or cookies.

Of the 5.75% reported for post launch activities, 1.87% was for continuous analysis of the web site, 2.34% was for re-development/improvement of the web site and 1.54% was for new/value-added online functions.

DISCUSSION AND CONCLUSIONS Non-Response Bias

We took extensive steps to try to improve the response rate, including attempting to telephone 200 of the survey non-respondents (twice!) to encourage their response and/or to gather information about why they did not respond. Various reasons were given, including “too busy”, “not relevant” (to them), “not interested”, and “confidential”. A high percentage (30.5%) said that they still intended to reply, but never did. Another 24% were not reachable by telephone to follow-up. If this last group was removed from the sample, the response rate would actually be 12%, but this would still be very low. The question then remaining is what sort of non-response bias there might be. If we look at the reasons given, we have a clue in the reasons “not relevant” or “not interested”. In our opinion, it is likely that non-respondents were likely to have chosen not to respond precisely because they didn’t use any WISDM and didn’t think them relevant. If we take this intuition into consideration, the percentages of the sample using or having tried WISDMs are likely to be even lower (possibly much lower) overall than the percentage of respondents. If we assume that all of the non-respondents (other than “confidential” or “not contactable”) did not use a WISDM, the percentage using at least one of the WISDM for which we surveyed could be as low as 1.15% and the percentage having tried one could be as low as 2.31%.

Non-disclosure due to “confidential information” is the only potential non-response bias that we could see increasing the percentages. Two (less than 1% of) non-respondents indicated that answers would be disclosing confidential information. If these non-respondents used methodologies, they would only slightly increase (some of) the percentages.

Web Consulting Organisations’ Low Methodology Usage

From the demographic data above, you can see that we are dealing with very small organisations here, which generally work on very small projects. Of course, these kinds of small consulting organisations provide very valuable services for other SMEs (their clients). As one would expect, such small consulting organisations do not have the ability (or perhaps the inclination) to research the literature of WIS development and to quickly come up with new ways to do their work in response to research developments. It is therefore likely that technology transfer issues are particularly relevant to such small consulting firms.

It is also likely that the relatively small average size of the WISs developed by these consultants (as opposed to WISs for larger client organisations) is less likely to benefit from using formal development methodologies. It may also be that the methods embedded within existing web development tools may be useful and appropriate for such small consulting organisations. However, the lack of use of a formal development methodology would explain the reported low quality of existing WIS. A possible strategy to improve technology transfer of WISDMs would be to embed them into commercial web development tools.

Another point to be made is that this study provides no evidence of differences between developers of large WIS and those of smaller WIS. Further research is needed into larger WIS development organisations, their clients, and their larger WISD practices and needs.

5

John R Venable & Fion C B Lim

Web Consulting Organisations’ Emphasis on Analysis

At face value, the percentages of time/effort spent on analysis are lower than those of design and implementation. However, 16+% is still a significant figure. Furthermore, if one considers the small and presumably more simple and straightforward nature of the WISs developed by the survey respondents, one would expect that more cursory analysis might be appropriate. However, this conclusion cannot be made in our opinion without further, more detailed research into the quality of the systems built and their resulting contribution to the organisation. Critique of Response Rate

The disappointing response rate makes the results of this survey much less (if at all) useful than it might have been. In our opinion, the survey suffered from several problems that have likely influenced the response rate. Firstly, the survey was seven pages long, which made it difficult for busy people. One reason for the length is that an attempt was made to combine two related student surveys into one (the other area was closely related on WIS development tools and their usage). The survey was also long because the topic was rather complex and dealt with a number of detailed matters. In retrospect, surveying different samples with several different, shorter, and more focussed surveys would likely have improved the response rate. However, this would also have limited the ability to compare responses to different questions by respondent. Such surveys might also have over-simplified the topic. Secondly, the quality of the list of recipients identified could have been improved and steps taken to identify and deliver the survey to a particular individual by carefully contacting people before sending out the survey. However, this would have been very resource intensive. Finally, if the survey were administered by telephone or face-to-face, the response rate might have been further improved. However, it is still likely that potential survey recipients would decline to be interviewed, which should still be considered as a non-response and potential source of bias. Nonetheless, we believe that some or all of the above suggestions could be useful for further research. Additionally, the use of case studies would be useful to investigate issues in more detail, but would not lead to the ability to generalise widely.

ACKNOWLEDGEMENTS

The authors would like to thank Dr. Donald McDermid and Dr. Christian Bauer for their assistance in formulating the research topic and method and Ms. Widya Kurniwan for her assistance in conducting the survey. We would also like to thank the anonymous reviewers for their feedback and suggestions. REFERENCES

Bauer, C. (1998) Using Reference Models to Develop WWW-Based Applications, submitted to ACIS

1999 (unpublished). Bichler, M. and Nusser, S. (1996) Modular Design of Complex Web-Applications with SHDT,

[Online] http://wicky-wu-wien.ac.at/w3dt/wetice/wetice.html [accessed 9 Sep 1999, appears to be no longer available]. Grønbæk, K. and Trigg, R. (1996) Toward a Dexter-based Model for Open Hypermedia: Unifying

embedded references and link objects, Proceedings of the 7th ACM Conference on Hypertext. March 16-20. Halasz, F. and Schwartz, M. (1994) The Dexter Hypertext Reference Model, Communications of the

ACM, Vol. 37, No. 2, Feb.. Isakowitz, T. (1997) What is RMM?, [Online]

http://wwww.stern.nyu.edu/~tisakowi/rmm/what_is_Rmm.html [appears to be no longer available]. Isakowitz, T., Kamis, A. and Koufaris, M. (1997) Extending the Capabilities of RMM: Russian Dolls

and Hypertext, Proceedings of the 30th Annual Hawaii International Conference on Systems Sciences.

6

Development Activities and Methodology Usage by Australian Web Site Consultants

Isakowitz, T., Bieber,M. and Vitali, F. (1998) Web Information Systems, Communications of the

ACM, Vol. 41, No. 7, July, pp. 78-80. Jayaratna, N. (1994) Understanding and Evaluating Methodologies: NIMSAD, A Systemic

Framework. McGraw-Hill, Sydney. Jones, D. (1999) A trip report from the first ICSE Web Engineering Workshop, [Online] http://cq-pan.cqu.edu.au [accessed 21 Apr 2000]. Lee, H., Lee, C. W., and Yoo, C. S. (1999) A Scenario-Based Object-Oriented Hypermedia Design

Methodology, Information and Management. Vol. 36, pp. 121-138. Lima, F. A. de and Eng, R. T. P. (1998) Towards an Integrated Design Methodology for Internet-based Information Systems, Proceedings of the Fifth International Workshop on Engineering Hypertext Functionality into Future Information Systems. April 20. Lowe, D. and Webby, R. (1998) Hypermedia Process Modelling, Internal report, University of

Technology, Sydney, [Online] http://www.eng.uts.edu.au/~dbl/papers/98ht01.pdf [accessed 14 October 2001]. Neuman, W. L. (1997) Social Research Methods: Qualitative and Quantitative Methods. (3rd ed.)

Allyn and Bacon. Schwabe, D. and Rossi, G. (1995) The Object-Oriented Hypermedia Design Model (OOHDM),

[Online] http://www.telemidia.puc-rio.br/oohdm/oohdm.html [accessed 14 October 2001]. Stamper, R. (1988) Analysing the Cultural Impact of a System, International Journal of Management.

Vol. 8, No. 3. Takahashi, K. and Liang, E. (1997) Analysis and Design of Web-based Information Systems,

Electronic Proceedings of the Sixth International WWW Conference. Santa Clara, CA, USA.

7