• Twitter
  • Linkedin
  • Facebook
  • Youtube
  • Gplus
  • Rss
  • Register
  • Log In
  • FREE Trial
  • subscribe to news
Sketch Engine
  • Home
  • News & Events
    • News
    • Blog
    • Events
    • Subscribe to email updates
  • Get Sketch Engine
  • User Guide
    • Quick Start Guide
    • User guide
    • Corpora
    • Glossary
    • Documentation
  • FAQ
  • About us
    • Lexical Computing
      • The company
      • The people
    • Research
      • Referencing Sketch Engine and bibliography
      • Research Agenda
      • Statistics in Sketch Engine
  • Contact
    • Contact us
    • Subscribe to email updates
  • Search
  • Menu
Blog - Latest News
You are here: Home / tokenization

tokenization

28th May 2016/in /by Michal Cukr

Tokenization is the automatic process of separating text into tokens.

Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on Google+
  • Share on Linkedin
  • Share by Mail
https://www.sketchengine.co.uk/wp-content/uploads/SE_logo_330x150-bleed-transp-bg.png 0 0 Michal Cukr https://www.sketchengine.co.uk/wp-content/uploads/SE_logo_330x150-bleed-transp-bg.png Michal Cukr2016-05-28 12:17:382016-12-13 13:29:33tokenization

Connect with us!

Newsletter

News

  • New English corpus from the Web 4th April 2018
  • POS tags 27th March 2018
  • Sketch Engine calendar 2018 – April 19th March 2018
  • New version of Danish corpus from the web 1st March 2018
  • A new Belarusian corpus (beTenTen) 26th February 2018

Archives

  • April 2018 (1)
  • March 2018 (3)
  • February 2018 (2)
  • January 2018 (3)
  • December 2017 (3)
  • November 2017 (3)
  • October 2017 (5)
  • September 2017 (1)
  • August 2017 (3)
  • July 2017 (1)
  • June 2017 (4)
  • May 2017 (3)
  • April 2017 (2)
  • March 2017 (6)
  • February 2017 (2)
  • January 2017 (1)
  • December 2016 (5)
  • November 2016 (1)
  • October 2016 (1)
  • August 2016 (1)
  • April 2016 (2)
  • February 2016 (2)
  • January 2016 (3)
  • December 2015 (1)
  • November 2015 (2)
  • October 2015 (4)
  • September 2015 (4)
  • August 2015 (5)
  • June 2015 (4)
  • May 2015 (1)
  • April 2015 (1)
  • March 2015 (1)
  • February 2015 (1)
  • January 2015 (2)

for learners of languages

Logo of SkELL – a tool for language learners and teachers

A Course in Lexicography and Lexical Computing

Logo of Lexicom – a workshop in corpus linguistics and lexicography
Cambridge, UK, September 2018
registration

term extraction

Logo of OneClick Terms – term extraction tool

Latest News

  • English Web 2015 - text statisticsNew English corpus from the Web4th April 2018 - 5:05 PM
  • Concordance from enTenTen13 English corpusPOS tags27th March 2018 - 6:13 PM
  • Sketch Engine calendar 2018Sketch Engine calendar 2018 – April19th March 2018 - 5:49 PM
© Copyright - Lexical Computing CZ s.r.o.
  • Twitter
  • Linkedin
  • Facebook
  • Youtube
  • Gplus
  • Rss
  • Home
  • News & Events
  • Get Sketch Engine
  • User Guide
  • FAQ
  • About us
  • Contact
token text type
Scroll to top