bnWaC: Bengali corpus from the web
The Bangla web Corpus (bnWaC) is a Bengali corpus made up of texts collected from the Internet. The corpus was prepared by the Corpus factory method, a method for developing large general language corpora which can be applied to many languages (A. Kilgarriff et al 2010). The Bengali Web corpus contains 11.5 million words. The corpus texts are available with lemmatization and POS tagging.
Part-of-speech tagset
This Bangla corpus was processed by an annotation tool developed by Microsoft Research Lab – India using this Bengali part-of-speech tagset.
Tools work with the Bengali corpus
A complete set of Sketch Engine tools is available to work with this Bengali web corpus to generate:
- word sketch – Bengali collocations categorized by grammatical relations
- thesaurus – synonyms and similar words for every word
- keywords – terminology extraction of one-word units
- word lists – lists of Bengali nouns, verbs, adjectives etc. organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
version 2 (2017 April)
- applied new tokenizer and lemmatizer
- corpus tagged using an Annotation Tool developed by Microsoft Research Lab in India
(2012)
- shallow tagging (no part-of-speech classification)
- universal word sketch grammar
version 1 2010
- initial version
Bibliography
Indian Language Part-of-Speech Tagset: Bengali
Bali, Kalika, Monojit Choudhury, and Priyanka Biswas. Indian Language Part-of-Speech Tagset: Bengali LDC2010T16. Web Download. Philadelphia: Linguistic Data Consortium, 2010.
Web corpora
BARONI, Marco; KILGARRIFF, Adam. Large linguistically-processed web corpora for multiple languages. In: Proceedings of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations. Association for Computational Linguistics, 2006, pp. 87–90.
Kilgarriff, A., Reddy, S., Pomikálek, J., & Avinesh, P. V. S. A corpus factory for many languages. In LREC, May 2010.
Search the Bengali corpus
Sketch Engine offers a range of tools to work with this Bengali corpus of web texts.
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.
