Wmatrix corpus analysis and comparison tool

Wmatrix is a software tool for corpus analysis and comparison. It provides a web interface to the USAS and CLAWS corpus annotation tools, and standard corpus linguistic methodologies such as frequency lists and concordances. It also extends the keywords method to key grammatical categories and key semantic domains.

Wmatrix allows the user to run these tools via a web browser such as Chrome, Firefox or Internet Explorer, and so will run on any computer (Mac, Windows, Linux, Unix) with a web browser and a network connection. Wmatrix was initially developed by Paul Rayson in the REVERE project, extended and applied to corpus linguistics during PhD work and is still being updated regularly. Earlier versions were available for Unix via terminal-based command line access (tmatrix) and Unix via Xwindows (Xmatrix), but these only offer retrieval of text pre-annotated with USAS and CLAWS.

Sections in this introduction to Wmatrix: screenshots, screencasts (short video introductions), acknowledgements and references for Wmatrix, and example applications and publications.

Tutorial for Wmatrix: with step-by-step instructions on how to compare Liberal Democrat and Labour Party Manifestos for the 2005 UK General Election (updated February 2010). Further examples of the application to the 2010 general election manifestos can be seen on Paul's blog. The plain text versions of the 2010 UK election manifestos can be downloaded for use in your favourite text analysis software (with thanks to Martin Wynne for editing two of the files). TEI encoded versions of the 2010 election manifestos are now available (with thanks to Lou Burnard).

Access the tool online at http://ucrel.lancs.ac.uk/wmatrix3.html (Wmatrix2 has now been retired). Wmatrix3 runs in Lancaster University's cloud infrastructure.

Usernames for Wmatrix are free to members of Lancaster University. If you would like access to Wmatrix, please contact Paul Rayson.

Usernames for academic research and teaching: (non-Lancaster academics) A free one-month trial is available for individual academic users, please contact Paul Rayson to set up a username and password. Please note that should apply from an academic email address. If you are a student, please apply via your course lecturer or supervisor. Once the one-month trial has expired, usernames are available for £50 per username per year from the online secure order page run by Lancaster University. Multiple usernames (or years) may be purchased at a reduced cost. Please ask Paul for details. Further development and external availability of Wmatrix currently depends on licensing its use.


Introduction to Wmatrix

Folders

Folders

Wmatrix users can upload their own corpus data to the system, so that it can be automatically annotated and viewed within the web browser. Each file is stored in a folder (equivalent to a folder in Windows or directory on Unix).

Input format guidelines

The analysis may be improved with some pre-editing of the input text, although pre-editing is not normally required. There are guidelines provided for texts to be tagged by CLAWS. Most important is the replacement of less-than (<) and greater-than (>) characters by the corresponding SGML entity references (&lt;) and (&gt;) respectively. The text may contain well-formed HTML, SGML or XML tags. If the text contains less-than or greater-than symbols in formulae, for example, then CLAWS may mistake large quantities of the following text for SGML tags, or fail to POS tag the file. The guidelines mention start and end text markers, but these are not required since they are inserted for you by Wmatrix.
Tag wizard

Tag wizard

Wmatrix users can upload their file and complete the automatic tagging process by clicking on the tag wizard. Once the file has been uploaded to the web server, it is POS tagged by CLAWS and semantically tagged by USAS. This process can be carried out step by step starting with the 'load file without tagging' option in the advanced interface. As a shortcut you can simply upload frequency profiles if you have them. The format for a frequency list is a very simple two column format with a total line at the head of the file. You can see an example of this. The column widths are not significant.

My Tag Wizard

My Tag Wizard is a variant of the tag wizard which allows you to override or extend the system dictionaries for your own data. There are two main uses. First, you can override the current most likely tag for any word or MWE. Second, you can extend the dictionaries in terms of coverage of vocabulary and tagset. For example, you can create a new tag by listing the words and MWEs that you wish to be tagged with it.
One workarea

Viewing folders

By clicking on the folder name, the user can see its contents. Following the application of the tag wizard, the folder contains the original text, POS and semantically tagged versions of that text, and a set of frequency profiles.

Simple and advanced interfaces

The user can toggle between simple and advanced interfaces in Wmatrix. The advanced interface offers more options and more control over the data.
Frequency list

Frequency profiles

From the folder view, the user can click on a frequency list to see the most frequent items in their corpus. Frequency lists are available for words in the simple interface, and in the advanced interface for POS tags and semantic tags. The lists can be sorted alphabetically or by frequency.
Concordance

Concordances

From the frequency list view, the user can click on 'concordance' and see standard concordances. These can show the usual word based concordance as well as all occurrences for words in one POS or semantic category.
Concordance

Key words, key POS and key domains: comparison of frequency lists

From the folder view, the user can click on compare frequency list to perform a comparison of the frequency list for their corpus against another larger normative corpus such as the BNC sampler, or against another of their own texts (once that text has been loaded into Wmatrix). This comparison can be carried out at the word level to see keywords, or at the POS (in the advanced interface), or at the semantic level (to see key concepts or domains). The log-likelihood statistic is employed by Wmatrix. For more details, see the log-likelihood calculator. In the simple interface, word and tag clouds are shown which visualise the more significant differences in the larger font sizes. In the advanced interface more detailed frequency information is also displayed in table form. Then the key comparison shows the most significant key items towards the top of the list since the result is sorted on the LL (log-likelihood) field which shows how significant the difference is. You should just look at items with a '+' code since this shows overuse in your text as compared to the standard English corpora. To be statistically significant you should look at items with a LL value over about 7, since 6.63 is the cut-off for 99% confidence of significance.

N-grams and c-grams

Recurrent sequences of words are called n-grams in Wmatrix. These are similar to clusters in WordSmith and lexical bundles in Biber's work. You can calculate n-grams of length 2 to 5 for each text. Collapsed-grams (or c-grams) are a merged version of these lists. They show you which 2-grams are subsets of 3-grams, which 3-grams are subsets of 4-grams, and so on. The resulting c-gram list is a tree structure with the longest n-grams on the left and shortest n-grams on the right.
Collocation

Collocations

Collocations in Wmatrix are pairs of words that occur together more often than would be expected due to chance. There are a choice of 11 different statistics that can be used to calculate the strength of association between the two words. For further details about these statistics, see the following paper:

Piao, S. (2002) Word alignment in English-Chinese parallel corpora. Literary and linguistic computing, 17 (2), 207-230. doi:10.1093/llc/17.2.207

The collocation feature was introduced in September 2009 and is currently in beta testing.

Screencasts:

This section shows short video introductions to the Wmatrix software. Further videos will be appearing soon.

Acknowledgements and references:

Wmatrix was initially developed within the REVERE project (REVerse Engineering of Requirements) funded by the EPSRC, project number GR/MO4846.

Lancaster University Proof of concept funding in July 2006 provided support for a new server and continued software development. In December 2006, further interface design using XHTML/CSS was carried out by Andrew Foote (InfoLab21 Knowledge Business Centre) funded under support from the European Regional Development Fund. Through a Lancaster University small grant (Towards an Online Conceptual Database of the Latin Vulgate Bible) a 'reader' interface is being developed for pre-tagged corpora.

Why the name, Wmatrix? Originally, I wrote a piece of software called Matrix which presented tables of frequency information from corpora, hence the named is partially derived from mathematical 'matrices'. This was Unix terminal based using 'curses'. I then wrote an X-windows version with a graphical user interface and named it Xmatrix. The web based version came next, hence Wmatrix. I also have a Java API to the website called Jmatrix. There's a note in my PhD saying that it has nothing to do with any films featuring Keanu Reeves, but if you're a Doctor Who fan like me, you may recognise another meaning of the Matrix.

The collocation feature in Wmatrix uses software derived from MLCT developed by Scott Piao.

The C-grams feature uses software developed by Andrew Stone.

Thanks are due to Steve Wattam who ported the semantic tagger, frequency profiling and concordance software to Linux from Solaris.

Please reference Wmatrix as one of the following:
Rayson, P. (2008). From key words to key semantic domains. International Journal of Corpus Linguistics. 13:4 pp. 519-549. DOI: 10.1075/ijcl.13.4.06ray
Rayson, P. (2009) Wmatrix: a web-based corpus processing environment, Computing Department, Lancaster University. http://ucrel.lancs.ac.uk/wmatrix/
Rayson, P. (2003). Matrix: A statistical method and software tool for linguistic analysis through corpus comparison. Ph.D. thesis, Lancaster University. (abstract or full text PDF version Postscript version )

Publications and applications using Wmatrix:

Wmatrix has been applied to numerous issues including: systems engineering, Aspect oriented requirements engineering, impact analysis of academic research, Ontology learning, Frequency profile comparison of written and spoken English, Political science research, Corpus stylistics, Training chatbots: comparison of human-human and human-machine dialogues, Key word analysis, Key word-class analysis for EAP, Key domain analysis, Phraseology, Comparison of political party manifestos, Metaphors in political discourse, Analysis of online language, Discourse analysis, e-learning materials development, modality, Computer content analysis: analysis of interview transcripts and Entrepreneurship studies and knowledge transfer.
  1. Abu Shawar, Bayan; Atwell, Eric. Using dialogue corpora to train a chatbot. In Archer, D, Rayson, P, Wilson, A & McEnery, T (editors) Proceedings of CL2003: International Conference on Corpus Linguistics, pp. 681-690 Lancaster University. 2003.
  2. Archer, D., Culpeper, J. and Rayson, P. (2005) Love - a familiar or a devil? An exploration of key domains in Shakespeare’s Comedies and Tragedies. Presented at the AHRC ICT Methods Network Expert Seminar on Linguistics. Lancaster University, 8 September 2005.
  3. Beigman Klebanov, B., Diermeier, D., and Beigman, E. 2008. Automatic annotation of semantic fields for political science research. Journal of Language Technology and Politics 5(1):95-120. http://www.cs.huji.ac.il/~beata/publications.html
  4. Calvo Maturana, Ma del Coral. 2012. Maternidad y voces poéticas en 'The Adoption Papers' de Jackie Kay: un estudio de estilistica de corpus. [Motherhood and poetic voices in 'The Adoption Papers' by Jackie Kay: a corpus stylistics study] PhD. Granada: Universidad de Granada.
  5. Calzada Pérez, Maria. 2010. "Learning from Obama and Clinton: Using individuals' corpora in the language classroom". Moreno Jaen et al. (eds) Exploring New Paths in Language Pedagogy, London: Equinox. p. 191- 212.
  6. Castaneda, A., & Lopez de D'Amico, R. 2012 PODER Y LENGUAJE EN BRUISED HIBISCUS, DE ELIZABETH NUNEZ: ANÃLISIS LITERARIO A TRAVÉS DE LA HERRAMIENTA INFORMÃTICA WMATRIX. [Power and Language in Elizabeth Nunez's Bruised Hibiscus: a literary analysis through the use of WMatrix] Tonos Digital [Online] 22:0. Available at http://www.tonosdigital.es/ojs/index.php/tonos/article/view/736/512
  7. Jonathan Charteris-Black and Clive Seale. (2010). Gender and the language of illness. Basingstoke: Palgrave Macmillan.
  8. Charteris-Black, J., & Seale, C. (2013). Men and emotion talk: Evidence from the experience of illness. Gender And Language, 1(1). Retrieved 1 May, 2013, from https://www.equinoxpub.com/journals/index.php/GL/article/view/17190
  9. Chitchyan, R., Sampaio, A., Rashid, A. and Rayson, P. (2006). Evaluating EA-Miner: Are Early Aspect Mining Techniques Effective? In proceedings of Towards Evaluation of Aspect Mining (TEAM 2006). Workshop Co-located with ECOOP 2006, European Conference on Object-Oriented Programming, 20th edition, July 3-7, Nantes, France, pp. 5-8. PDF version
  10. Da Silva AL, Dennick R. Corpus analysis of problem based learning transcripts : an exploratory study. Medical education. 2010;44(3):280-8.
  11. Da Silva AL, Dennick R. 2009 CORPORA ANALYSIS OF PROBLEM-BASED LEARNING TRANSCRIPTS. In ASME Annual Scientific Meeting 2009. Edinburgh, UK
  12. Da Silva AL, Dennick R. 2009 - PBL - "it's all talk". Corpora Analysis of Problem Based Learning transcripts. In o Association for Medical Education in Europe (AMEE) conference 2009. Malaga, Spain
  13. Da Silva AL, Dennick R. 2010 -Applying corpora research methods to the study of Language and Clinical Reasoning in a Problem Based Learning Curricula. In Promoting Excellence in Healthcare Educational Research - A Multiprofessional Conference. Law and Social Sciences Building University of Nottingham, Nottingham
  14. Da Silva AL, Dennick R 2010 EVALUATING PROBLEM-BASED LEARNING TRANSCRIPTS USING CORPUS ANALYSIS: DO MEN AND MACHINES AGREE?. In 14th Ottawa Conference. Miami, Florida, US
  15. Da Silva AL, Dennick R 2010 EVALUATING PROBLEM Corpus Analysis of Problem-Based Learning Transcripts: A new method to look into PBL. In o Researching Medical Education. London, UK
  16. Da Silva, Wharrad & Pitt., 2011. Interprofessional Learning Sets: Exploratory analysis of online students discussions (Poster). In NET Conference, 2011. Cambridge, UK.
  17. Da Silva, & Pitt., 2011. More than words: Analysis of students' Interprofessional online discussions. In EIPEN 2011. Ghent, Belgium.
  18. Marilyn Deegan, Harold Short, Dawn Archer, Paul Baker, Tony McEnery, Paul Rayson (2004) Computational Linguistics Meets Metadata, or the Automatic Extraction of Key Words from Full Text Content. RLG Diginews, Vol. 8, No. 2. ISSN 1093-5371.
  19. Demjén, Z. (2011) The role of second person narration in representing mental states in Sylvia PlathÕs Smith Journal. Journal of Literary Semantics. 40(1), pp1-22.
  20. Doherty, N., Lockett, N., Rayson, P. and Riley, S. (2006). Electronic-CRM: a simple sales tool or facilitator of relationship marketing? 29th Institute for Small Business & Entrepreneurship Conference. International Entrepreneurship - from local to global enterprise creation and development. 31 October - 2 November 2006, Cardiff-Caerdydd, UK.
  21. Gabrielatos, C. and McEnery, T. (2005). Epistemic modality in MA dissertations. In. Fuertes Olivera, P.A. (ed.) Lengua y Sociedad: Investigaciones recientes en lingüística aplicada. Lingüística y Filología no. 61. Valladolid: Universidad de Valladolid, pp. 311-331. PDF version
  22. Gacitua, R., Sawyer, P., Rayson, P. (2008). A flexible framework to experiment with ontology learning techniques. In Knowledge-Based Systems, 21, 3, April 2008, pp. 192-199. DOI: 10.1016/j.knosys.2007.11.009
  23. Yufang Ho. (2007) Investigating the key concept differences between the two editions of John Fowles's The Magus - a corpus semantic approach.? The 27th International Conference of the Poetics and Linguistics Association (PALA), Kansai Gaidai University, Hirakata, Osaka, Japan, 31 July - 4 August 2007.
  24. Xin Huang (2003) A Computer-aided Diachronic Content Analysis of Twentieth Century Political Discourse in China. MA dissertation in Language Studies, Lancaster University.
  25. Jones, M., Rayson, P. and Leech, G. (2004) Key category analysis of a spoken corpus for EAP. Presented at The 2nd Inter-Varietal Applied Corpus Studies (IVACS) International Conference on "Analyzing Discourse in Context" The Graduate School of Education, Queen’s University, Belfast, Northern Ireland, 25 - 26 June, 2004. PDF version
  26. Emilie L'Hote and Maarten Lemmens (2009) Reframing treason: metaphors of change and progress in new Labour discourse. CogniTextes, Volume 3, http://cognitextes.revues.org/index248.html
  27. Leech, G., Rayson, P., and Wilson, A. (2001). Word Frequencies in Written and Spoken English: based on the British National Corpus. Longman, London. (see the companion website for more details)
  28. Patrick Maiwald (2011). Exploring a Corpus of George MacDonald's Fiction. North Wind: Journal of George MacDonald Studies 30: 50-84. Available here: http://www.snc.edu/english/documents/North_Wind/By_genre_or_topic/Language/Exploring_a_Corpus_of_George_MacDonald%27s_Fiction_-_Patrick_Maiwald.pdf
  29. McIntyre, D. and Walker, B. (2010) 'How can corpora be used to explore the language of poetry and drama?' in McCarthy, M. and OÕKeefe, A. (eds) The Routledge Handbook of Corpus Linguistics. Abingdon: Routledge.
  30. Afida Mohamad Ali (2007). Semantic fields of problem in business English: Malaysian and British journalistic business texts. Corpora, 2, 2, pp. 211-239.
  31. Murphy, S. (2007). Now I am alone: A corpus stylistic approach to Shakespearian soliloquies. Papers from the Lancaster University Postgraduate Conference in Linguistics & Language Teaching, Vol. 1. Papers from LAEL PG 2006 Edited by Costas Gabrielatos, Richard Slessor & J.W. Unger. PDF version
  32. Nakano, T. and Koyama, Y. (2005). e-Learning Materials Development Based on Abstract Analysis Using Web Tools. Knowledge-Based Intelligent Information and Engineering Systems. 9th International Conference, KES 2005, Melbourne, Australia, September 14-16, 2005, Proceedings, Part I, LNCS 3681, Springer, pp. 794-800. DOI 10.1007/11552413_113
  33. O'Halloran, K.A. (2011a) 'Limitations of the logico-rhetorical module: Inconsistency in argument, online discussion forums and Electronic Deconstruction', Discourse Studies, 13(6): 797-806.
  34. O'Halloran, K.A. (2011b) 'Investigating Argumentation in Reading Groups: Combining Manual Qualitative Coding and Automated Corpus Analysis Tools', Applied Linguistics 32(2): 172-196.
  35. O'Halloran, K.A. (2010) 'Critical reading of a text through its electronic supplement', Digital Culture and Education, 2(2): 210-229. http://www.digitalcultureandeducation.com/cms/wp-content/uploads/2011/06/DCE1022_ohalloran_2010.pdf
  36. O'Halloran, K. (2012) Deleuze, Guattari and the use of web-based corpora for facilitating critical analysis of public sphere arguments. Discourse, Context & Media. Volume 2, Issue 1, March 2013, Pages 40-51, ISSN 2211-6958, 10.1016/j.dcm.2012.12.001.
  37. Vincent B.Y. Ooi, Peter K.W. Tan & Andy K.L. Chiang (2007) Analyzing personal weblogs in Singapore English: the Wmatrix approach. Studies in Variation, Contacts and Change in English. Volume 2. Research Unit for Variation, Contacts and Change in English (VARIENG), University of Helsinki. http://www.helsinki.fi/varieng/journal/volumes/02/ooi_et_al/
  38. Vincent B.Y. Ooi (2008) lexis of electronic gaming on the Web: a Sinclairian approach, International Journal of Lexicography, 21 (3), 311-323. doi: 10.1093/ijl/ecn021
  39. Magali Paquot, Sylviane Granger, Paul Rayson and Cédrick Fairon (2004) Extraction of multi-word units from EFL and native English corpora: The phraseology of the verb 'make'. Presented at Europhras, European Society of Phraseology, 26-29 August 2004, Basel, Switzerland.
  40. Potts, A. and Baker, P. (2013) Does semantic tagging identify cultural change in British and American English?, International Journal of Corpus Linguistics 17(3): 295-324.
  41. Paul Rayson (2004). Keywords are not enough. Invited talk for JAECS (Japan Association for English Corpus Studies) at Chuo University, Tokyo, Japan, 27th November 2004. (PDF versionslides)
  42. Rayson, P. and Smith, N. (2006) The key domain method for the study of language varieties. The Third Inter-Varietal Applied Corpus Studies (IVACS) group International Conference on "LANGUAGE AT THE INTERFACE". University of Nottingham, UK, 23-24 June 2006. PDF version
  43. Sawyer, P., Rayson, P. and Cosh, K. (2005) Shallow Knowledge as an Aid to Deep Understanding in Early Phase Requirements Engineering. IEEE Transactions on Software Engineering. Volume 31, number 11, November, 2005, pp. 969 - 981. ISSN 0098-5589.
    doi: http://doi.ieeecomputersociety.org/10.1109/TSE.2005.129
  44. Sera, H. (2012). Dickens' 'The Signal-Man' and Poe's 'The Fall of the House of Usher': How did they describe terror? Presented at PALA 2012, Malta.
  45. Shapero, J. J. (2011). The Language of Suicide Notes. Unpublished Thesis. The University of Birmingham. http://etheses.bham.ac.uk/1525/
  46. Shapero, J. J. & Blackwell, Susan A. (2012) "'There are letters for you all on the sideboard': what can linguists learn from multiple suicide-note writers?" p.225-244. In Samuel Tomblin, Nicci MacLeod, Rui Sousa-Silva and Malcolm Coulthard (Eds.) Proceedings of The International Association of Forensic Linguists' Tenth Biennial Conference. Centre for Forensic Linguistics, Aston University, U.K. [ISBN: 978 1 85449 432 0] www.forensiclinguistics.net/iafl-10-proceedings.pdf
  47. Francois Taiani, Paul Grace, Geoff Coulson and Gordon Blair (2008) Past and future of reflective middleware: Towards a corpus-based impact analysis. The 7th Workshop On Adaptive And Reflective Middleware (ARM'08) December 1st 2008, Leuven, Belgium, collocated with Middleware 2008.
  48. Walker, B. (2010) Wmatrix, key-concepts and the narrators in Julian Barnes' Talking It Over. In Busse, B. and McIntyre, D. (eds.) Language and Style, pp. 364-387.
  49. Walker, B. (2012). Character and Characterisation in Julian Barnes' Talking It Over: A Corpus Stylistic Analysis. PhD Thesis, Lancaster University.
  50. Walkerdine, J. and Rayson, P. (2004) P2P-4-DL: Digital Library over Peer-to-Peer. In Caronni G., Weiler N., Shahmehri N. (eds.) Proceedings of Fourth IEEE International Conference on Peer-to-Peer Computing (PSP2004) 25-27 August 2004, Zurich, Switzerland. IEEE Computer Society Press, pp. 264-265. ISBN 0-7695-2156-8. PDF version
  51. A number of papers were presented at the PALA 2007 conference (29-30 July 2007, Kansai Gaidai University, Osaka, Japan) including those by Geoffrey Leech, Yu-fang Ho, Dan McIntyre, Haruko Sera, Brian Walker. Mick Short and Brian Walker also ran a Workshop: Using Wmatrix to compare scenes from Harold Pinter's Betrayal. See the book of abstracts on the conference website for more details.
  52. EPSRC InfoLab21 Knowledge Transfer Study Report and the ICT Knowledge Transfer Research Project