Once we have decided on the topic of our research, we need to ask ourselves a question: What do I need to know and why? Only then we can decide on the best way to collect information and what we will do with it.
There are different types of data: quantitative, qualitative, case study, action research and so on. Surveys will generally be the method for quantitative data. Case studies are considered qualitative.
You need to make decision on which method is the best for your work.
The method chosen will be highly influenced by the amount of time you have and the funds available. Efforts should be made to cross check findings. This is called triangulation. It is a technique that checks the validity of data by collecting it from two or more sources. We combine several research methods and explore the same concept or issue. We compare one finding to another. This can be time consuming.
Any data collected should always be examined critically. It also needs to be reliable. The procedure needs to collect similar results. A question that produce different answer is unreliable. Questions that ask for opinions may produce different answers as people have different experiences that affect their opinions.
Reliability
We can check reliability through test-retest method. It happens by administrating the same test after some time after the first. There is also alternate form method, where equivalent versions of the same item are given and results correlated. The split-halve method is where the item in the test are split into two matched halves and scores then correlated. There are, however, some disadvantages of those methods.
Validating
Validating the findings is another complex problem. These findings can be vague and have many unanswered questions. An explanation considers whether the research methods leads to data that support the conclusions the researches draws from it. The researcher needs to be aware of conclusions that are not substantiated from the data. Measuring the extend of validity can have many variations. We might not be involved in complex testing and measurement for our projects, thus we make efforts here to examine the findings critically.
To do this, we might ask ourselves whether another researcher using the same research instrument would be likely to get the same or similar responses. It might be helpful to tell other people what you are trying to find out to measure and check the questions you devised are likely to do the job. We need to achieve some degree of reliability and validity in the question.
You might be able to use a software package to analyse the responses. Here, we need to be mindful of the questions we ask as they may be difficult to analyse. The most often software package used to analyse data is IBM's SPSS Statistics (Statistical Product and Service Solution). This software is expensive but the data can be presented in number of ways, including charts and graphs. Many businesses have a data and research advisors. The way we word our questions may influence the type of analysis.
In small studies, it might be better to keep manual methods of data analysis and interpretation of results. It is about the careful selection of the topic and being sure about the purpose of the study, collecting and interpreting results and producing a well-written report on time.
There are no set rules of how many surveys or interviews should be conducted. Your aim is to obtain as representative number of responses as possible to enable to fulfil the objectives of study and answer the study questions. It is important to select the best tool for the job.
Analysis of documentary evidence
Your research evidence may be used to supplement other research findings and form the basis for your research. It is important to evaluate it objectively and avoid bias.
Most projects will require analysis of documentary evidence. We need help to locate, categorise, select and analyse documents. In some projects, documentary evidence is used to supplement information obtained from other methods. For example to check reliability obtained from surveys or interviews. In some project it will be the central and exclusive method of research. Documentary analysis can be extremely valuable alternative source of data.
During the document search it is helpful to clarify exactly what types of document exist. Research can involve the analysis of images, films, videos, or other non-written sources. The most common documents are, however, printed and manuscript sources. Generally records are kept in electronic form. Sources can be qualitative or statistical. It is important to employ the critical method of analysis to check how the figures have been produced. What has been counted, by whom, why, when etc.
The two different approaches
When analysing documentary evidence we can use source-oriented approach. Here, we allow the nature of the sources determine the project and ask questions for the research. Full investigation of material would lead to investigation of the area. Rather then bringing the predetermined questions we are led by the material they contain.
The second and more common method is the problem-oriented approach. This involves formulating questions by using other research methods and then by reading secondary sources. This method investigate what has already been discovered about the subject before establishing the focus of the study and then researching the relevant primary source. As the research progress it is clearer which sources are relevant and more questions will occur.
The documents location
Document searches need to be conducted in the same way as literature searches. Such sources might be published or unpublished. The documents may include government papers, guidance papers, statistics, reports, policy papers as well as local sources. Internet is a great place to search for official documents, but we also must be prepared to search for local sources. Some sources may be confidential so we need to obtain permissions to access them.
At the local level, the nature of the project will lead us to a particular source. In some industries there is a strong emphasis on evidence-based research, importance of document and personal responsibility of staff for records they create or use in the course of their duties. NHS, for example, develops the code of practice for handling documents and each medical organisation has to have a policy statement on how records are managed. This can be very helpful for researches. Organisations are encouraged to establish an inventory of records that can help researches. There are administrative and personal records that may exist. Minutes of meetings, memos, policy statements exist.
Organisations also provide detailed policies on the retention of records and minimum retention dates. Different projects will require different searches. It is important to ask what archives are stored by individuals, departments and organisations. Researches might be frustrated by the different policies as some sources might be destructed.
Primary and secondary sources
Documents can be provided by primary and secondary sources.
Primary sources are those that came into existence in the period under research. Secondary sources are events of that period based on primary sources. The distinction is complicated. Some sources might be primary from one point of view and secondary from another.
Primary source can be divided into:
* Deliberate sources - produced for the attention of the future researches. These include autobiographies, diaries or letters intended for further publications. They involve deliberate attempt to preserve evidence for the future.
* Inadvertent sources - used by researches for some purpose other that that for which they were originally intended. For example, the records of legislative bodies, agencies and local authorities, national databases, inspection reports, newspapers and journals, examination papers, personal files, budget statements, policy statements, research project records.
Such inadvertent sources are generally more valuable types of primary sources.
"Witting" and "unwitting" evidence
Witting evidence is the information that the original author wanted to impart ( make known). Unwitting evidence is everything else that can be learned from the document. Often underlying assumptions made or unrevealed from a document. All documents provide "unwitting" evidence and it is the task for the researcher to determine its significance.
Content analysis
The number of documents we can study will be influenced by the time we have for our research. Normally, it is not possible to analyse everything and we must make the selection. Try not to include too many deliberate sources. If the time runs short, try and reduce your selection.
The proper selection is important for content analysis. It is defined as "systematic technique for compressing many words of text into a fewer aligned with explicit rules of coding" ( Stemler , 2001). It is a technique to objectively identify specified characteristics of a message. It assesses the meaning and significance of the source. Content analysis can be used for different purposes. For example it is useful to analysing the significance of speeches and articles by political readers.
The first steps
The best approach with content analysis will start with research question and then decides on the sampling strategy after the "coding" or "recoding units" has been defined. There are different ways of defining such units, but essentially, these are sampling units, such as policy statements, which are then analysed in sentences using categories, such as the purpose of the policies or the advise given.
Content analysis could develop "emergent coding" with categories being formulated as the sources are explored. Or they can be prior coding based on the previous reading and the generation of theoretical ideas. We might be looking for theme or phrases. Theme is a recurring pattern that focuses on the key concept. There could also be sub-themes developed. A sub-theme should be used sparingly (in small quantities).
The approach generally involves counting the number of times particular terms occur in sample of source, but also number of columns devoted to the subject or number of photographs in a publication. The words mentioned most often, however, might not reflect greatest concerns.
The chosen sampling method is very important. It needs to be replicable and justified. It might be possible to study all the documents in a particular category, but in other cases sampling technique is needed. You must be able to justify the sample and it must be sufficiently large to allow valid conclusions. You must also establish the frequency of chosen terms and place them in context before explaining and interpreting.
After analysing word frequency it is valuable to assess the context of the key words to test consistency.
Critical analysis of documents
The analysis of documents can be a subject to internal and external criticism.
External criticism aims to discover whether a document is both genuine and authentic.
Whether it is what it says its purpose is and truthfully reports on the subject. For example, an observer could write a report of a meeting that he never attended. The report would be genuine, but not authentic because he was not present at the meeting.
For the external criticism is is important to know that the author wrote the document. The questions could include the following:
* Was the author known to be in place from which it came at the time it was supposed to be written?
* Do other sources collaborate that the person wrote it? Is the document consistent with all the other facts known by the author?
* Does it use the same structure and have the same form?
* Is it typical for other documents written by the author?
An effort must be made to determine whether the author actually composed the document.
Internal criticism is most likely to be used in small-scale research. The contents of a document are subjected to rigorous analysis. We seen answers to the following questions:
* What kind of document it is? How many copies are there?
* What does it actually say? Are the terms employed as you would generally use them? Special legal papers might use specialised language which must be mastered.
* Who produced it? What is known about the author?
* What was its purpose? Did the author aim to inform, command, remind. A document is always written for a particular readers. The reader should try to be aware of the purposes and intentions of the author during the reading.
* When and in what circumstances was it produced?
* Is it a typical or exceptional of its type?
* Is it complete? Has it been edited?
* What is known about the authors social background, political, views, aims and past experience?
* Did the author experience or observed what is being described?
* Did the author tell the truth or exaggerate? Distort or omit?
* How long after the event did the author produce the document?
All these questions aim at critical analysis. Examine sources carefully. People may indicate prejudice to ignore proposed change. Decide whether any political reasons may influence the tone or emphasis of the work. Assessment of reliability must include questions:
* Reliable for what? Is it a reliable explanation of the authors views on the issue?
* Is it representative of those views?
* Is it truthful?
Fact or bias?
One important aim of an assessment is to decide whether the paper states facts or bias. Writers will rarely declare their assumptions. It is the aim of the researcher to expose them. Look out for any terms that suggest partisanship (prejudice in favour of particular case, bias). Ask yourself whether the evidence supplied convincingly support the arguments.
If the document goes against the author's own interests, it might increase the likelihood that it tells the truth.
If you detect bias, it does not mean that the document should be dismissed. Conclusions can still be drawn from "unwitting" testimony. Often, political processes influence innovation. The bias document will need to be analysed cautiously and compared with other sources.
Try to put yourself in the position of the author to see through their eyes. Instead of jumping to early conclusions, deliberately seek contrary evidence to test the truthfulness and watch out for your own bias. It is easier to recognise bias in others then ourselves.
Sources can be interpreted in different ways but the guiding principle is that everything should be questioned. Scepticism and empathy both need to be employed.
It might be that the techniques of critical analysis are simply common sense. It might be true, but as we study a document we gain "higher common sense" with fuller appreciation of the evidence.
Eventually the critical method becomes a habit.
If your research requires to source documents from different sources, you need to ensure it is analysed objectively.
Checklist:
Analysing documented evidence can be a challenging task. I hope that the above guidelines can help you through the process. Whether you are a marketing professional, quality executive, auditor or a business analyst, a thorough evidence review is essential for the purpose of your research.
Below are the key steps:
1. You must decide first how you will use the document data. As a supplementary or exclusive method of gathering data.
2. Choose the approach to the documents, either through predefined research question or formulated after the research is done.
3. Undertake the document search to ascertain which will give you the answers.
4. Analyse the nature of the source: deliberate or inadvertent.
5. If document is large, decide on sampling strategy.
6. There might be different types of evidence in different documents. Look for "witting" and "unwitting" evidence.
7. Compare the documents with other sources.
8. Ask further questions about the author.
9. Look for bias.
10. Decide whether document is reliable for the purpose.
11 . Gain full appreciation of the document value, gain insights.
12. Make the critical method a habit.
Bowen, G.A. (2009) Document analysis as a qualitative research method, Qualitative Research Journal, 9 (2): 27–40.
Comments